Part 3: Web Analytics: Deep dive

In the preceding parts we have scratched the surface of web analytics, looked at a methodology for systematic work and also some project theory by discussing the impact of style guides and design on the analysis work.

Now it is time to go through many of the things that most of us do not automatically associate with web analytics. Among other things, some of the tools that can be useful for working with follow-up, and not least a bunch of tips and anecdotes from reality.

When it comes to measurability in general, there are two main categories that metrics can be divided into (and, yes, you need to use both kinds):

• Quantitative data. Provides numbers on how the website performs, for example all the information about users as a group that you can extract from your statistics tool. These metrics answer the questions how, when and what about the users. What equipment they have, when they visit the website, where they came from and so on. Behavioural data that provides insight into the extent of something.
• Qualitative methods. Concerns how users experience the website, as they themselves would express it. This information is in the form of text rather than numbers. It can be answers to questions in a website survey, interviews, usability tests or features on the website where you let the user say what they thought about their experience or what they just read. Beyond people's attitude to a website, it can in some cases involve observing users so you can take the chance to ask follow-up questions, if needed. You are looking for answers to the question why. Qualitative information contributes the perspectives that quantitative data lacks.

Regarding qualitative surveys, it is wise to be attentive to the situation the user is in when they are to answer questions. You have surely encountered being offered to fill in a survey about how you experience a website. Sometimes these survey invitations pop up before you have even started using the website. The question then arises whether it does not affect how the respondent fills in the survey? Are their answers identical to visitors who have clicked around on 2–3 pages first? If you intend to draw conclusions based on this data, it is unwise to take chances with your data quality.

Image 37: This type of survey is directly inappropriate on a user's first visit to a website.

It can be wise to at least log how many page views the user managed before they filled in the survey. Whether it is a logged-in person or otherwise returning user is also good to know. For returning users, it is probably not the end of the world to throw up the survey early – they have at some point previously experienced the website. Here you reuse your user segmentations. Perhaps you discover some new interesting groupings based on the metadata you can collect about the respondent, such as where they came from, whether there are geographical differences, etc.

It is difficult to evaluate your own website yourself

One thing that seems easy to forget is that we who create, own, develop or otherwise are responsible for a website are really not representative users. We are also not particularly good at imitating real users since we all come with entirely our own perceptions and prior knowledge. If you have been involved, it is perhaps primarily your prior knowledge that makes you a poor tester.

That is why you need to regularly conduct usability tests. Within what is called universal design, sometimes inclusive design or design for all, a list³⁶ has been developed of what usable design is about, namely:

1. Equitable use – the use should be fair and possible for everyone despite our varying abilities.
2. Flexibility in use – versatility in use.
3. Simple and intuitive – simple and self-explanatory.
4. Perceptible information – content and information should be possible to perceive.
5. Tolerance for error – error-tolerant and forgiving construction.
6. Low physical effort – should require little exertion.
7. Size and space for approach and use – there should be room for use and ease of doing the right thing.

These points are not specific to digital creations but can definitely be applied to how you design websites. They were created in 1997 by The Center for Universal Design at North Carolina State University, so they were compiled long before the web started looking like it does today.

Starting a web project or a redesign

There is of course an enormous amount to think through when you are about to start a web project, regardless of whether it is the content you are reviewing, some new design that needs to happen or whether it is a fresh start with web analytics. A prerequisite for being able to understand what is worth analysing is that you know what on a website is, or should become, what lives up to the purpose of the website.

Suggestions for things to think through early include:

1. List which parts are most important, important, normal or unimportant. Not everything can be important for all intended target groups, all the time.
2. Which parts of the website support the most important activities? These pages need to be analysed in the hunt for improvement opportunities. Such as their placement of buttons, comprehensible texts, how easy it is to find your way there, etc.
3. How easy is it to navigate between the different activities? Think through the navigation structure without preconceptions. Is it for example possible to find your way back to each starting point if you received a direct link via email? Is it easy to switch from the activity of signing up for the newsletter to correcting incorrect address details?
4. How easy is it for users to get through an activity? That a shop's survival depends on customers being able to pay is quite obvious. On a website, you need to ensure there are as few and low barriers as possible to completing an activity all the way.
5. What constitutes a successful visit to the website? What things need to have occurred for you to objectively point to a user's session on the website and say it was a successful visit? Signed up for a newsletter? Downloaded a whitepaper, read a news article and disappeared? Think through which metrics indicate a successful visit.

Material that does not relate to the above points is usually distractions and not interesting to improve. Rather, it should probably be removed, or at least hidden; users should be led around it and it should be marked as potential junk for your own search engine. Feel free to think through what characterises a valuable page, such as whether a page without a call to action can be considered valuable and if so why. Set the goal that new web pages should have a call to action or be closely tied to a business goal.

So what is a call to action (CTA)? Well, it is the design element on a web page that the user is intended to interact with. It can be a button, a link or whatever is the reason the page exists. A CTA is the way you as the creator of a website try to guide the user through an activity towards a goal – a completion.

You are meant to look critically at all your website's subpages. Would it not feel good if you could explain what the majority of pages on the website contribute?

What content is good (and what can be discarded)?

So what is good content on an existing website and can you find out what works? If you work with web analytics, you need to form an opinion about what the quality factors are and how you can work with measurability to get to know your users and their behaviour.

Working with web analytics is much more than diving into website statistics; there are masses of other tools that can give a helicopter perspective of your web communication. But for it to be meaningful to analyse, you first need to define what is good – good content, good user behaviour, good quality and what is good for the business. Otherwise, you risk working on improving material that has no future prospects. But do not despair; there are ways to find out what is demanded by users. Through search analytics, among other things.

Something that has been neglected for a very long time, or perhaps simply not been got around to, is evaluating whether the search function you offer works as it should. Sometimes the search function is mainly a source of frustration. The question is whether something so vital on a website should be allowed to remain if you do not follow up whether it contributes benefit or is only a source of trouble. Search analytics is our next deep dive.

Search analytics is more important than some think

Today you can safely assume that every website or intranet has a search function. Some organisations have even taken the step to building something of an internal Google, or Enterprise Search as it is usually called. Where I work, we have invested heavily in this technology for many years.

Reflecting on and analysing the use of your own search function is what goes by the term search analytics (or Site Search Analytics in English). It is about taking part in the words used by users and what behaviour they have. What is most commonly sought, when they cannot find their way and so on.

As you have surely figured out, this is not the same thing as SEO (search engine optimisation), but there are some similarities between wanting to reach out on others' search engines and trying to optimise the use of your own search function. Search analytics is perhaps more related to web analytics than to SEO. Many of the web analytics tools I have seen actually have functions for doing basic search analytics. For example, in Google Analytics you can configure your site search to take advantage of the other tools offered also for how the website's search works.

If you instead choose to steer your own destiny, it is entirely possible to process the search logs. That is, the text files where all search queries are recorded by the system. Then it is only your own imagination and budget that set the limits.

Search analytics provides a unique insight into the users' experience

What truly makes search analytics unique is that you get an insight into what users are after. They even write it with a series of words about what they are looking for. That type of insight you can only dream of achieving when it comes to the behavioural data that other web analytics largely consists of. Because you get so many words, it is a mix of both quantitative and qualitative investigation. Both volume and the personal.

Motivation for working with search analytics can include supporting your design decisions. For example, because you can show what things people search for on the website. That you also have data to make the search function better does not hurt, and as we know, the search function never gets better than the content it highlights. Working with search analytics can reveal what needs to be improved in the website's content, what is missing, and much more.

How it works

First and foremost, through a visit to a search function, the visit is usually captured by the web analytics tool used on the website. On the public web, it is very common for this tool to be Google Analytics, but there are also a number of other tools with corresponding functions. One aimed at organisations that are cautious about letting others access their users' browsing patterns is called Matomo. The advantage and disadvantage of Matomo is that you can have it locally in your own technical environment; no one else needs to find out what visitors your website or search function has. Or what they search for.

Besides the visit to the website itself, one or more things can happen once users perform a search. Firstly, if you have configured your web analytics tool correctly, it will capture the search term, but it is also common for the search function to write down the search query to a log file together with some information about the user. This extra information most likely concerns date, time, the user's IP address, and perhaps some information about what kind of equipment was used. Read up on user-agent³⁷ if this interests you.

All this data can then be analysed independently from your regular web analytics tool, if you have that type of search function, regardless of the functions that already exist in the web analytics tool.

Search analytics risks becoming technical

As you notice, words like “log”, “user agent” and so on are used in this context. It is indeed the case that your ability to access the search log may depend on the IT department's goodwill and interest. Historically, data like these have been collected for use by technicians to troubleshoot technical problems, and they have probably not thought of sharing. You may be the first to ask for access, at least as someone who does not work in IT. The reaction to the question can surely vary.

One way to get the discussion going can be to give them a present in the form of a good new tool. Say that IT gets help with the financing of a multi-competence tool, both for business intelligence and as it happens also search analytics? There are several products on the market that can help with search analytics. For those of us sold on open source, the so-called ELK stack³⁸ stands out from the crowd. It is a combination of Elasticsearch, Logstash and Kibana. That is, a search engine, log analysis software and visualisation in one package.

Image 38: ELK can look like Region Västra Götaland's search editor portal. Here the search term “lediga jobb” (job vacancies) is inspected and appears to show a seasonal variation.

ELK is a service you install on your servers. If you want a less ambitious alternative, I think you can check out a self-help programme for data analysis and visualisation. One of my favourites is Tableau which happens to come in a version that is free to use, Tableau Public³⁹. You can install it on your own computer to inspect log files or data sources in general.

Search analytics in content and design work

As mentioned earlier, you can get good help with design decisions. Not least because even more data about the user's experience helps in the argumentation, but also because you get something of a cheat sheet in discussions about how things should be named on the website. It can become obvious how things should work or how real live users name things. Here is a tool for meeting the constantly higher demand for evidence and for organisations wanting to be data-driven. A word I learned when I worked as a developer at Sahlgrenska University Hospital was vederhäftighet (reliability/trustworthiness), meaning something's degree of dependability. At such a workplace, it is not news that you should be able to back up your claims with reliable data; that is what research is about – if I may be allowed to greatly simplify things.

For those who work with content, it can be nice to know which words your users know, what they search for and so on. It helps to have a list of sought-after words when you are writing headlines, choosing which synonyms to sprinkle into the introduction, etc.

Something else that search analytics and editorial search work can contribute to content work is keeping on top of things, working strategically with your content. Among other things, there is a method for finding what you want to weed out, thanks to the search engine. This is done by discarding material that is ROT, meaning Redundant, Outdated or Trivial. This type of content costs money either by trying to maintain it or when a user actually uses it. Not least, it is often in the way in the search function. ROT content will always mess things up regardless of how well you build your own search function, and it can steal attention on external search engines from the content you would rather highlight.

How meaningful is content no one finds or has use for? Such things you should be able to discard. If the search function knows about content that has no designated owner, that too is probably a signal that no one would miss the material.

The relationship to search engine optimisation and click statistics

I have many times joked, in contexts around search engine optimisation, about the difficulty of selling product names. Like ‘Windcracker 3000 GLX’ when people google search terms describing what they are after, like ‘rain jacket for running’. In search analytics, semantically enriched information is hidden. You can see which words were used but ideally also how search queries were reformulated in the hope of getting better results. It is a fantastic source for how you conduct your SEO work.

Search analytics should be at least as important as checking how users behave on the website. To be completely honest, what the web statistics system collects is not much more than a logged stream of clicks on various things. While search analytics can approach qualitative investigations about what users were actually looking for, not which path they clicked along while searching.

Forget the long tail, settle for the short head (for now)

If you are familiar with the concept of the long tail⁴⁰, which aims to describe variation, you have realised that certain search queries are extremely common while you quickly end up on the rarities. The majority of search queries are what is called onesies and twosies, meaning the queries far out in the tail, the search queries that are only asked once or twice.

Image 39: The short head and the long tail ≈ a few search terms are enormously common.

It can be good to know that the long-tail distribution also is often called a Zipf curve, the 80/20 rule and surely more names I have not yet heard. They all describe the variation within a volume of something. In this case, that search queries quickly start to show a pattern of recurring queries (short head) but also uniqueness (long tail).

I have the fortune of knowing a mathematician; among other things, he works calculating insurance premiums. Often when I have a difficult question, I take the opportunity to bounce it off him. Including in this case:

To be perhaps even clearer: a Zipf curve shows the distribution of a set's elements in descending order. If you want to be pedantic, the frequencies are normalised so that the frequencies sum to 1, which means it is a probability distribution that is being described.
- my friend Morgan Skogh

In brief: it is surprisingly common for a user to search for precisely the most popular search terms.

The 100 most common searches are vastly more common for users to search for compared to the searches that are 100–200 most common. It is likely that the fourteen most common search queries account for a full 10% of all search queries.

What should you do with this knowledge? If you start by quickly looking at the hundred most common search queries and ensure there are good answers, you will have ensured that the search function works okay for a fairly large proportion of users. The more you work with this, the smaller the gain from the improvements becomes. Imagine if all work were so easy to explain what constitutes a reasonable level of effort.

What you are looking for in basic search analytics is patterns. Patterns that provide insight into how things work, or finding patterns of systematic problems you can correct.

A usage pattern you need to know about is which the most common search queries are. These can change depending on the time of year, week or day. These insights are something that can be converted into opportunities in how you work with your content. For example, getting a signal about when the seasonal variation of looking for a new job has started, when people have started searching for norovirus, or whatever content is demanded by your users.

Segment – if possible

What do the users in your most important segment, the ones most valuable to you, search for? To be able to know this, your logs may need to be supplemented, or if you can take advantage of advanced filtering in your web analytics tool.

Segmentation is about getting to know your users and their behaviour. It can prove very valuable also when it comes to web analytics. Since language is such a natural factor in search analytics, due to users' word choices, the most obvious segmentations are things connected to users' language. It does not need to be different languages as in what mother tongue they have; it can just as well be that you find lay language versus technical jargon.

Examples of segments that can be interesting for getting to know your search function include:

• What is the traffic source? Which can be translated to which domain name, if you have several, where the user started their search. Think campaign site versus the regular website.
• Is it a new search user or a returning one?
• Loyalty and frequency of returning.
• Conversions. Among which users do conversions occur, and what distinguishes them from those where nothing of interest happens? In what way does the search function contribute to the sessions that convert?
• Average value. For those of you conducting commerce or who can translate a visitor's activation of a goal into actual money, it is interesting to segment out the most profitable, the normal ones and compare them with each other and with those who are not profitable. Are there lessons?
• Geography is a classic. Not infrequently there are differences depending on where users are. Perhaps there is something in their experience you can improve?
• Timing, meaning when during the day does goal fulfilment happen? Perhaps it is a certain time of the week, month or year. It is important that both the website and search function are ready for this and contribute to a positive experience.

(Have a search editor...) Keep an eye on zero results

A recurring task for the person responsible for the search function is to check which search queries users have asked when there were no results to show. The reasons for no results being shown can be several, but the user does not care about that. According to the user, the search function has failed. Except for those searching who do not want any results (like searching among patents, trademarks you want to register, etc.), zero results are a gigantic failure. Zero results are unfortunately quite common when searching within websites or individual web applications.

Trying to keep track of zero results, and checking that there is good material for the most popular queries, is what you should have a search editor for. As the name suggests, it is a person who through editorial measures ensures that the search function works well. The search editor often needs permissions and the trust to remedy problems with content others have created, at least if they are to be effective.

The problems I have stumbled upon for visitors getting zero results have always been one of these:

1. There is no relevant material that matches the search query.
2. The material exists but is called something other than what the search query is looking for.
3. The material exists, but the search function has an automatic filter that means the user's search query is only run against a subset of what the search engine knows about.

Not all visitors' search queries are always relevant to the website's goals, so keep an eye on the queries that matter. Which search queries matter is a question you can only get an answer to by doing the groundwork of formulating goals for the website.

If there should be content that matches the search query, it needs to be within the search editor's mandate to fix it. Sometimes it is as simple as adding a few keywords to the material that should be shown. Other times it can become very advanced. Like when synonym management is needed, or if the content is in a data source that the search engine does not have access to. My book Webbstrategi för alla has an entire chapter on information architecture, where you will find tips on how to tackle this type of problem.

That book also covers some self-evident things, one might think, about how an error page should be designed. When it comes to informing a user that no results were found, it can be done in a more engaging way than the text “Sorry, no results”. The zero results page should be helpful. This can be highlighting common search queries, offering guesses about spellings, etc.

“Why not do like Google?”

Those who work with search and search analytics regularly hear ungrateful (and uninformed) comments that they should get their act together. How hard can it be when Google does that search thing so well?! There is actually a good answer to that.

Enterprise data simply isn't like web or consumer data – it's characterised by rarity and unconnectedness rather than popularity and context.
– Charlie Hull, flax⁴¹

The short answer is that the information you have on your website, or within the organisation, is not comparable to what Google does. Google operates on a public web where billions of pages exist, those pages link to each other, they compete to reach out on search engines, etc. Hands up the web editor who has followed up how well their material has reached out on the organisation's own search function. Hello, is this mic on? Sarcasm aside, it is probably not very common, yet.

Even the users you have on your own search function differ markedly from those who go to Google. The users you have on your website have already found their way; they give your search function their trust. Those using Google are more open to suggestions about who can provide the best material. Your user believes they can find something they think you have. Not least – they have given your website the trust to solve their need! That trust you need to manage.

Key performance indicators for evaluating the search function

Besides the fact that you have probably already figured out that the number of search users who avoid seeing a zero result can be a good key performance indicator, there are a few more suggestions I can offer. If you want to standardise questions and be able to compare your answers with other search functions, check out Net Promoter Score (NPS); there is a bit about this in this book. Then users should answer 0–10 on the question of how likely it is that they would recommend the search function to a friend or colleague.

Try to avoid immediately ending up measuring saved time as a key performance indicator. It is difficult to argue what business benefit a few seconds here or there provides. It leans a bit towards vanity metrics to keep track of how large a proportion has searched and clicks on something in the results list, similarly with logging whether they click on the top three results. These values help you get to know your search users, but they are not worthy goals to steer towards.

Just as you need to work on setting measurable goals for the rest of the website, you can and should weave in the search function. Which parts was the user on before they completed a defined goal on the website? Was the search function part of that process?

If you want examples of search-specific metrics and key performance indicators, here are some:

• Proportion of queries that return zero results. Work towards a low proportion, at least in the segments of users you are targeting. A micro goal is to try to avoid giving a bad experience, which a zero result is.
• Proportion of search queries or sessions that lead to a click. Meaning something in the search results was clicked. A micro goal that the search function contributes to the user finding something worth clicking on.
• Proportion of queries that lead to the visitor leaving the website. A metric, not a goal, to see which search terms fail to retain the search user's interest. Corresponds to bounce rate in other web analytics.
• Which search queries are used in sessions where the user converted. A metric rather than a goal.
• Which are the pages on the website where the most users start their search. A typical metric for trying to find content that may need to be improved or included in an A/B test to evaluate whether you can influence the experience in the right direction.
• Goal: Conversion rate among those who have used the search function. Tells when the search function has been part of a session that created value. Can be difficult as it depends on highlighting content where conversion can occur, but interesting as a segment for exploring what within search creates value.

If you want to do an even deeper dive into search analytics, I can recommend the book Search Analytics for Your Site⁴² by Louis Rosenfeld. That was the book I read to cherry-pick the best parts for what you just read. Something I have not mentioned that he writes well about is an evaluation model of an existing search function. If you are facing the task of being able to prove that search performance has improved over time, you have yet another reason to read that book.

What is in the web analytics toolbox?

A widespread misconception, or perhaps exaggerated simplification, is that web analytics is about going through collected website statistics. Of course, that work is part of working with web analytics, but you should be aware that you are then looking at statistics based on how a website is and not what it should be. In some sense, you deserve what the statistics show.

Quantitative tools

Almost too obvious to need mentioning, but the most common quantitative tool in web analytics is all our solutions for checking website statistics. The largest market share in that market is held by the freemium tool Google Analytics. Yes, Google Analytics costs money if your website is large enough. Then there is a long list of other tools with a mere few percent market share, like Adobe Analytics and Matomo. We have covered that type of tool in the first part of the book, and if you are missing something, the internet has no shortage of either tips or books to buy on this topic. In this part of the book, we shall instead focus on less common tools. Be prepared for some appetisers.

Tools for content analysis

Search consultants like to call this index analysis. It can actually be a good reminder for those of us who have a search engine, that the search engine has knowledge about what the website contains and this can be used for follow-up work. But there is more within content analysis that is not quite as natural to lump together with the requirement of having your own search engine.

Things search engines already know about your content include:

• Which words are on the pages. You can use this to keep track of the words the organisation has decided should not be used. Compare with the New York Times' list “Words we don't say”⁴³, but also useful for seeing if a term you want to be found on is missing.
• Whether page title lengths are within good SEO practice, say under 70 characters? The search engine could list the longest page titles as a report.
• Cross-referencing whether commonly searched words appear in page titles, headings or structured HTML markup. Useful for finding untapped SEO potential, not least for external search engines.
• Following up what type of material is published. For those making an effort to build a better mobile experience, it is interesting to check that the trend of published PDF documents is declining.

You get some of these tools for free if you work with search analytics for your own search engine. If you want to see corresponding figures for Google, there is an ever-growing toolbox on their Search Console (formerly Webmaster Tools) and Bing has something similar.

Image 40: Google Search Console shows which words it frequently finds on my book blog.

Furthermore, (your own) search engine's methods for collecting content for its index can surely be supplemented so you do not need to produce yet another tool that crawls around your website. In its simplest form, it is developing reports for when the search engine encounters potential problems. Like how often, or on which subpages, it encounters 404 error pages. Or if it encounters a page with identical content to something at another address.

Content analysis encompasses more than search techniques, as the paid service Siteimprove demonstrates. They catch broken links, spelling errors and even carry out some accessibility tests. For those of you who do not have an ambitious search engine strategy, Siteimprove can possibly be a solution for keeping track of the linguistic aspects, so the editorial style and brand presentation follow the communications policy you have.

Image 41: Google reports what usability problems it has found in the content of a website.

Last but not least within content analysis is to cross-reference the statistics on what gets visits according to your web statistics with everything the search engine knows exists. What has not been used in 12 months perhaps you should automatically delete or archive?

Log analysis

Analysing logs once again has the future ahead of it. Before tools like Google Analytics broke through in the mid-2000s, it was various log tools many of us had at our disposal. The difference between Google Analytics and a log-based tool is how they collect their data. Google Analytics has historically been based on being close to the user interface and keeping track of usage as a third party involved in the communication. Either you have tracked users through small JavaScript code snippets, but some similar tools have used transparent small images as an alternative method.

Logs, on the other hand, are collected without involving third parties. Almost every technical device or piece of software writes a log file behind the scenes. A bit like a diary to show to developers or maintenance technicians when they are looking for the cause of something.

The reason I believe log analysis will once again become interesting for more and more people is because many have started to realise that users selectively block content. Most probably to protect their privacy. Logs are not something a user can block; they are handled within the service being used.

Just as mentioned in the section on search analytics, these logs can be processed by software that creates added value. Suddenly you can analyse things other than the strictly introverted technical aspects. You can gain insight into what software is doing, what service it provides users.

If you want to do this on a slightly larger scale, there are tools like Splunk and Logstash. Splunk is software that charges based on the amount of information you read in. It can help you decrypt the logs, find patterns and relationships. Information modelling, quite simply.

Logstash, on the other hand, as part of the ELK stack, Elasticsearch-Logstash-Kibana, is an open tool whose business model is to sell support to larger organisations. The product itself is free, but it can take a lot of energy to get started.

The advantages of Splunk and Logstash are that they are productised. Then they become more comprehensible from an IT buyer's perspective and there is support available to purchase. Both solutions cost a fair amount of money, but in different ways. Using Splunk costs as you get used to it and see the benefit, while Logstash requires a more active IT management and the expenses come before the benefit is obvious.

Now there is an ever-growing crowd of competitors to the IT department owning the question of BI (Business Intelligence). These tools focus on the digitally mature knowledge worker being able to help themselves. Tableau is a hyped such tool that focuses on self-service for inspecting more or less large data sets. Like any classic software, it is installed on a computer and then you read in a log file (or other structured information). After that, you just start exploring the data source. Preferably through visualisation, which makes deviations stand out. A dedicated BI consultant I often encounter in these circles usually refers to Tableau as a Photoshop for data, with the point about its simplicity in showing something.

What I myself have found enormously useful about Tableau is for quickly looking at a log file from several perspectives. For example, a log file with a few hundred thousand rows of “slow” database queries from a website that was extremely dependent on the database for presenting content. I had set it so the database server wrote down a database query to a log file if the query took longer than 0.5 seconds to run. I had three different types of data in the log:

1. The time when the database query was posed.
2. How long the database query took to complete.
3. The database query itself, meaning which content was requested.

What I could do thanks to visualisation was to quite easily look at this otherwise rather unwieldy data set from several angles. I could quickly rule out that the time of day caused any slowness. However, there was a correlation between the time the query took to complete and which table in the database the query was directed at (which is apparent from the database query). The result was a very precise basis for which parts of the system had bottlenecks, and which parts had nothing to do with the matter at all.

Image 42: Sometimes the user does not even come into contact with the web server.

Unfortunately, you need to be alert to whether you have data somewhere that gives a fair picture of the situation. At my workplace, we have had a prolonged period of wavering around secured communication over the network (with SSL certificates). Our IT department has not been in agreement with the world about what a trustworthy certificate should look like, which has led to our users in some cases being prevented from reaching our services. In some cases, we have had data, for example the times users were stopped by our firewall.

Image 43: A firewall's primary purpose is not to help a web analyst, but it surely logs data you can use.

For us, the log file in the firewall was so extensive that it had to be reset at regular intervals. Beyond that, the file was so large that it would become a major project in itself to filter out a reasonable amount of content so it could be inspected in some software we were familiar with. At first glance, it was not worth the effort in this case.

Performance analysis

Image 44: To verify other performance data, you can compare with how quickly Google has indexed your website.

Reports for a website's performance can be obtained from several different sources. If you want a quantifiable overview, Google's Search Console is useful if you go to the Crawl Statistics report. But if you are prepared to pay, by all means check out Pingdom's services. The advantage of Pingdom is that they are specialised in performance monitoring and there you can specify from where the monitoring should be done. If you have a website that should provide service for a Nordic market, it is not that interesting how fast it goes for a visitor on the US west coast. The speed of light is admittedly fast, but in these contexts not fast enough. With Pingdom you can for each test choose whether the monitoring should be done from servers in Europe, North America or Asia.

Image 45: Pingdom shows uptime and response times for 10 websites if you pay.

A newer function in Pingdom is to get more details about the website's performance. In this case, you point out one or more of your most important pages and they are monitored at a time interval. This can be a good way to keep an eye on your most important landing pages, so that an unthinking web editor does not accidentally publish a too-high-resolution image, or that other careless mistakes are not addressed in a reasonable time.

Image 46: Pingdom reviews my book blog's start page.

The last quantitative performance tool I intend to recommend is that you who already have Google Analytics actually have a view for Google's tool PageSpeed Insights. There you find out how well-performing your website is, and it can be a good list for you who are just about to start with performance optimisation.

There is a long list of tools that may not fall under the quantifiable flag in the sense that you have a constantly ongoing data collection. But many of these tools can be integrated, or by a developer hacked together, so they become quantifiable and interesting in ways other than as manual spot checks. This is where you jump to the section on qualitative tools if you cannot be bothered with technicalities :)

Image 47: Apica checks how my small website handles 10 simultaneous users.

An interesting approach is to be proactive and check how your website is doing under load, to check whether it then shows weaknesses worth fixing. A company that does this is Apica⁴⁴ and they have the tool Loadtest which can simulate an influx of users to your website. This type of tool has IT staff as its primary target group, so they may already exist in your organisation, but I believe a normally talented web analyst can manage this type of solution. You will probably notice that the interfaces can be a bit unfinished for first-time users.

Another tool you probably have not missed if you follow my blog or have been to any of my lectures is Google's PageSpeed Insights⁴⁵, which can be used independently from Google Analytics. I have since 2010 used it to highlight the need to work with web performance, had top lists and all in all compared tens of thousands of websites against each other. What PageSpeed contributes is that it has an API that makes it possible to integrate your check, for example with your CMS. Design it so that when a page is created or changed, it is checked against a list of external APIs to see if it meets your performance budget. Or you do as I do and have semi-automatic runs of a set of pages, and when something less good is found, the relevant people are notified.

PageSpeed, in addition to performance factors for mobile and desktop, also has a number of usability factors it checks. Such as whether links are too closely placed, whether the content fits on the screen and a few more.

Image 48: Sitespeed.io is a helping hand for those who can handle a bit of technology.

A Swedish and open project that in many ways replaces PageSpeed Insights is sitespeed.io which by being a layer between you and the APIs checks the web page against both WebPagetest and PageSpeed Insights. For those of you who are curious, the project is of course on GitHub where you can download all their code⁴⁶.

Speaking of WebPagetest, which has already been mentioned for their visual comparison between two different websites, it is also an open project. With a little effort, you can have your own local environment for doing the same tests. This is good, for example, for those who want to work with intranet analysis, or if you want to run many more tests than the free services allow.

Qualitative tools and methods

For those of you who yawned through the difference between quantitative and qualitative, I can mention that qualitative tools and methods are what resemble spot checks. The kind of thing that does not impress those who work with statistics. But the benefit of qualitative work is to provide perspective and get answers to questions, not just knowing the extent of something. So it is without doubt important to also work with the qualitative.

Many of the quantitative tools have aspects that allow you to examine individual parts of something, meaning they too can be used qualitatively, but I did not intend to repeat them unnecessarily here.

Many probably think of user experience or service design when they hear ‘qualitative methods’ and that is not strange; it is a natural part of getting a handle on the situation. Typically, qualitative methods are associated with things like:

• Surveys to understand existing users.
• In-depth interviews to learn more about intended target groups, their needs, habits and preferences.
• Competitive analysis to see how you compare against someone else.
• Ethnographic studies where you observe users in their natural environment.
• Usability tests where you examine whether what has been constructed is usable, or what difficulties can be identified.

All of this can be done with a highly varying level of ambition. Despite this being a bit outside the book's topic, and not being my expertise, I still wanted to give some tips on what you can relatively easily solve yourself.

For the points listed above, there is no shortage of good literature, so I did not intend to attempt writing about them when others have already done it so well. Instead, I refer those of you who want a light-hearted learning about how to follow up and evaluate user experience to Steve Krug's book Rocket Surgery Made Easy⁴⁷. Those of you who prefer something considerably more academic can check out the book The Elements of User Experience: User-Centered Design for the Web and Beyond⁴⁸. Steve's book will give you a way of working that you dare try yourself, something you can start with at almost no cost and practise on colleagues before you tackle outsiders. The other book's strength is rather in its evidence for each method's merits. It can be good even for the impatient reader to skim through, to later use as a reference work.

Something that unites most of the above-mentioned qualitative methods is that they involve users in one way or another. It may seem like the only thing in the toolbox, but I would argue that is directly incorrect. That many of us have this picture is because it is precisely those parts that are comprehensible to package as an offering. Let us not be limited by what consultants feel comfortable offering and putting a price tag on. Let us instead be open to everything that might provide insight into individual users' experiences.

There is a long list of specialists you can benefit from for making a qualitative overview. People who with a little preparation can investigate qualitative aspects, do spot checks and provide feedback on potential improvements. Several of them you surely already have as colleagues. I myself, who view the web from a web developer's perspective, have a whole lot of tricks you are unlikely to find at a usability firm. I find it hard to think less of other professional groups.

RUM tools – show the users' actual experience

If you instead want slightly more reliable results you do not need to justify on your own, there is always RUM (Real User Monitoring) available. In some cases, it is a bit of an unholy mix between quantitative and qualitative, at least if you have masses of users. What RUM does is inspect real user sessions and display them for you, without you needing to gather the users or seek them out. Loosely put, you record the user's entire session with the help of various tools. Afterwards, you can then inspect that user session, filter among the collected user sessions in the hunt for something specific.

It can be an effective way to present your case to colleagues, leaders or clients. That users have actually had precisely that experience.

Image 49: Inspect anonymised users' sessions on the website.

In Google Analytics, these sessions are under Audience → User Explorer. There you can see some overview information about users that allows you to choose to inspect the experiences that led to conversion, or whatever you find interesting to investigate. Each user is de-identified and is instead represented by a Client ID.

By marking up your content with the metadata standard Schema.org, you can help Google understand what value each activity has. This can help when you want to compare converting users with each other. Perhaps there is something to improve?

Image 50: Report from a user's experience of buying a book.

Image 51: Hotjar tracks where the user's mouse cursor has moved.

Other tools include for example Hotjar⁴⁹ and Clicktale⁵⁰. Both have functions for visually recording the user's experience of a website. You can see exactly where the user paused, where the mouse cursor moves across the screen and how they navigate through the content and sub-steps that exist.

What makes this type of tool highly interesting is that they provide insight into what the user journey actually looks like on the way towards a goal. They give a slightly more qualitative angle on the otherwise so dominant conversion funnel. A conversion funnel gives you information about the quantity of converting users at each sub-step, while this gives you qualitative insight into what that experience can look like.

Pingdom, which we will talk more about in the performance section, also has a RUM tool. Roughly simplified, Pingdom measures how quickly a website loads, and normally they have a bunch of servers placed around the globe that do this for us. But that does not really provide qualitative data, only quantitative data, or even fictional. They have nothing to do with a real user's experience. What Pingdom's RUM tool can be good for is if you want to know how well your website loads for users on another continent. I, who have historically hosted my private web projects at Loopia in Västerås, should perhaps consider moving the content closer to the users. With the help of Pingdom's RUM, I can follow up whether my English blog is tolerable for real users connecting from the US west coast. It is a good tool for being able to act based on real data.

Image 52: With Pingdom's RUM tool, you can set which users you track and what threshold values you want to group the information into.

Technical quality during construction

Image 53: PageSpeed Insights gives reports on a page's performance and usability.

Both Google PageSpeed Insights and Sitespeed.io are viable also as qualitative tools when you want to do spot checks on performance and the technical environment. This may need to be done directly after deployment to ensure that expected quality has been achieved.

I hope the concept of accessibility does not come as a surprise to anyone. A lot of effort can be put into this type of work. Sometimes you are forced to spend a lot of time on it as it is increasingly being legislated about a minimum level. In the US there is Section 508⁵¹ about information and communication technology needing to be accessible. In 2015, Swedish anti-discrimination legislation was tightened and began to apply in digital contexts too; it thus became illegal not to offer a basic level of accessibility. At the time the book is being written, I have not been informed of any precedents about exactly how strictly the law is interpreted in different contexts, but it is worth monitoring Swedish practice a few years after a new law has been introduced. Beyond this, during 2016 an agreement⁵² was reached within the EU that public authorities' websites, intranets and apps should have good accessibility. Popularly called the Web Accessibility Directive.

To familiarise yourself with the topic, you can test your technical quality, meaning evaluate the code that lands in users' browsers. Both the web standards organisation W3C's accessibility standard WCAG and Swedish Webbriktlinjer.se offer a quick test⁵³ you can try. It is a bunch of questions to answer.

Image 54: There are images without correct handling of alt texts at gp.se

For a more general quality check, there are an infinite number of tools aimed at developers of various kinds. But one worth a look for anyone who just wants to do a health check on a page is the web agency Netrelations Inspector⁵⁴. It gives straight and comprehensible reports about accessibility, among other things.

Visual tests

Before mobile users' entry onto the web, web developers used to manually check how what they built looked in the 2–3 latest versions of Internet Explorer, the latest Firefox and Chrome. Nowadays, however, the variation is so enormous that it is not possible to test in that way. Nor is it feasible for most to buy equipment for testing as it constantly needs to be renewed.

Image 55: At Filament Group they have a so-called “device lab” to be able to try designs on different devices.

One difference from before smartphones was that the ambition then was to make websites “pixel perfect”. Meaning you sent long lists of requirements on how different things should be shifted a few pixels in one direction or the other. Pixel-pushing has fortunately been given up by most with the advent of responsive web design. Perhaps because people have understood that it is downright impossible to make it look identical on all devices.

One solution to this challenge is to take advantage of services like BrowserStack⁵⁵. They have equipment you can test-drive via the internet to know what a website looks like. At the time of writing, they have Windows and Mac computers, iOS and Android devices. Note that the respective device's actual browser is used; it is not a simulation.

BrowserStack can also be integrated into developer tools so the tests are handled a bit more automatically. Then you can have saved images to be able to review whether something has happened to important pages; you also get a log of what it has looked like historically. Visual tests are about important design elements being visible and having the conditions to work; it is not about monitoring the layout or details in visual design. You should be more interested in whether your call to action is gone than whether the logo is a pixel askew.

Now there are thousands of different combinations of devices and browsers. To start selecting from that jungle, one tip can be to check what it looks like on the devices that have the highest bounce rate on your website.

Image 56: From my Mac I borrow a Windows 7 computer over the internet to verify the design on my error page.

Create a test page to evaluate changes in web design

For it to be a fair comparison from one measurement to another, you need to test under the same conditions. In other words, you must set up a test page on your website and that page is the one you run tests against to know that the results are comparable over time. What can differ is the web server's response time, but the page's content in an editorial sense needs to be the same from one time to the next.

If you do it this way, you know whether changes to the website have made things better or worse. It can be that you try switching themes if you run WordPress, that your developers have upgraded something or possibly done new development.

How to design your test page

For your test page to be meaningful, its editorial design needs to remain the same over a longer period. If you are to be able to compare the result before an upgrade of the publishing system with after, the content needs to be the same.

My suggestion is that you create a subpage solely intended for testing. That page should contain text, images and things that are normally complex for a regular page on the website. This is not where you embed a bunch of strange external services, widgets, video clips or things that neither you nor the others involved have control over.

Do a baseline measurement

When you have your test page, you do a first measurement with the services you want to use, such as Google PageSpeed. Then you reconcile the result with everyone involved and document it somewhere (so people understand that this is what they have to live with going forward). In some suitably formal way, you should probably explain that it is not acceptable for the result to get worse over time; rather, it should improve.

At the next update or change, you need to check that it actually does not get worse; ideally it should get better, but definitely not worse. In other words, any external parties, such as consultants, should already in the assignment learn what acceptance criteria exist. For example, you could require in the assignment document that:

Since we currently have 67 out of 100 according to [your benchmark], and you as the contractor have accepted exceeding this level, it is to be regarded as a warranty matter at no cost to us as the client if the delivery on the established test page is not at least 70 out of 100 according to [your benchmark].

I am not a lawyer, but if the contractor fails with this, you should hopefully be able to skip paying the last invoice.

A few more perspectives on web analytics

First and foremost, as always, I want to point out that what you find in your web statistics system is probably only a subset of the data you need to do a proper job in web analytics. This insight you surely reach as soon as you think further about how you measure business goals on the website.

Key Performance Indicators (KPI)

Key performance indicators are not a concept exclusive to the web. Rather, it is an established concept for a long time within the business world. A key performance indicator in our context tells how the website contributes value to the organisation. It can be how a customer's activity on the website can be converted into pounds and pence, or how for example the organisation's reputation has been affected.

Many organisations already have key performance indicators. What you as a web analyst have to do is see if they can also be used on the web. It is an excellent way to go from fuzzy hobby activity to getting leaders within the organisation to understand what benefit the website contributes. It makes it easier to defend desired investments and is the common language you can use during elevator pitches with the top boss, as well as what you recommend for the annual report.

The business goals for the website

If there are no key performance indicators that can be reused (or easily modified for reuse on the web), you can hopefully find your business goals documented somewhere. If you happen to lack both business goals and key performance indicators applicable to the website, your problem falls a bit outside the scope of this book, but I would guess you are not alone. On the contrary, that is exactly what I have often heard when I have asked my clients the ultimate question: “Why do you have a website, actually?!” There is usually a whole lot of throat-clearing, looking downward, scraping feet against the floor and generally distracted behaviour. I understand that people are embarrassed, but I do not think it is that uncommon.

The truth is probably that many are right in the shift between having a website “because everyone else has one” and figuring out exactly what they want from their website. In many cases, those I have spoken with are in the middle of an organisational tree. Then the overarching goals are not always relevant for one's own website. At one's own level in the organisation, you do not always have established business goals, or awareness of them is not particularly high.

For government agencies and the public sector, in the worst case you can read the assignment document received from the state. There, the assignment is formulated in the worst conceivable bureaucratic language.

In the private sector, the latest annual report is surely a place to find business goals, but also more short-term aspirations. If you have a good business plan documented, the business's goals and value creation should be described there.

Examples of formulating key performance indicators

Examples of what you might want to measure with the help of a key performance indicator include something's impact on things like:

• Sales
• Profit margin
• Cost reduction
• Average level of customer satisfaction on a scale
• Maximum waiting time for a customer

When I did research on what a good key performance indicator might be, I found the book Key Performance Indicators (KPI): Developing, Implementing, and Using Winning KPIs. It argues that good key performance indicators have the following characteristics:

1. Not tightly tied to financial goals. It should not be expressed in currency terms. Here, just as in other web analytics, it is the trend or change that is interesting.
2. Well-timed. It should be possible to see how the key performance indicator performs more often than annually in the annual report. Ideally, you would want this divisible and comparing one hour with another, but at least every day or week needs to have a checkpoint for the key performance indicator's performance.
3. Decision-oriented focus. Business leaders should be able to act based on the key performance indicators.
4. Are comprehensible. All employees should understand the metric's benefit, be able to figure out how it is measured and in what way they can act for the result to improve.
5. Can be tied to specific groups. Responsibility should be assignable to a specific group of employees and this group should through their work be able to influence the key performance indicator's trend.
6. Has a tangible impact. This is where meaningfulness comes in, that the key performance indicator should make meaningful use in the business. For those of you working with balanced scorecards, the key performance indicator should affect more than a single scorecard perspective.
7. Has a limited unknown factor. A good key performance indicator encourages positive activities. A poorly thought-through key performance indicator, on the other hand, can lead to undesirable behaviour among employees, for example that customer support hard-sells instead of helping the customer with the reason for the call.

The key performance indicators can themselves be of different kinds, namely whether they are quantitative or qualitative. This is not a book about statistics, but some basic insights into what statisticians work with is healthy for any web analyst. This becomes a bit of a repetition, but also an expansion of qualitative versus quantitative.

Qualitative data, methods and information – data about feelings, attitudes, experiences and opinions

Qualitative data are those that do not directly appeal to statisticians; they are more of the unstructured kind that cannot easily be processed by computers. Examples of qualitative data are surveys, feedback from users, like-clicks and similar. Most often these are expressed in the way people themselves express themselves, in human language, meaning in a way that makes it difficult to interpret if you do not read and evaluate them.

Quantitative data and methods – data about usage and behaviour

Quantitative data are, as the name suggests, data that follow statistical models for how to describe something. The most common source in web analytics in this case is website statistics where you collect data about how users move around the website – their behavioural data as a group, that is. Then there can be other data sources that are affected by activity on the website, or conversely that activities outside the website affect the web. An example of the latter is sudden traffic spikes and to the extent you can figure out their cause. It can be completely normal seasonal variation, but it can be due to disease outbreaks, Midsummer weather and much else.

Standardised methods and metrics

To be able to compare yourself with other organisations, there is a point in trying to reuse others' measurable goals. One way is to check out websites where commonly used key performance indicators are listed; another is to use the same tools, or utilise existing methods for measurement. Then you also get help with how the measurement should be carried out.

CTR – Click Through Rate

CTR is something most people have probably heard of. It is about expressing in percentage how large a proportion were exposed to an offer (often called a call to action) and then clicked onward towards the shared goal. This offer is thus a path you can navigate by clicking on a button, or similar, to add a product to the shopping cart, sign up for a newsletter, etc. If a button has a CTR of 77%, it means 77% of those who saw it chose to click on it. A CTR of 4.1% in Google's search results means that 4.1% clicked on your particular result. What the others do cannot be read from a CTR; that is not what is measured. It also happens that on pages with multiple simultaneous offers, one speaks of a CTR as any of your offers getting a click.

Sometimes you hear people talk about what CTR someone has and the audience makes impressed sounds, or comments that “that was unusually good”. To be completely honest, I have not yet understood this myself, and from the literature I have read, there is no optimal CTR. Nor is it supposed to be meaningful to expect any particular CTR. This leans more than permissible towards measuring vain things. That a certain proportion of users clicked to the next step does not mean any business goal has been achieved. Like page view counts, it is an isolated metric that tells a story about usage, but the value of that knowledge is in itself extremely low other than for the person working with that specific website.

But with that said, it can be worth experimenting with your CTR to see if you can redirect users' attention, i.e. raise your CTR, and whether that ultimately has a positive end result. It is not certain that a raised CTR leads to higher conversion. It is the digital equivalent of a pushy salesperson. On the websites I have inspected, the path to a conversion is often long and winding. It is not self-evident that it works to try to shorten the time for the user's deliberation. However, if you manage to identify a user segment that you dare experiment with, then get started.

Ways to experiment with higher CTR:

• Work with A/B testing on your own website or in advertising contexts to figure out which gets the highest CTR. Beyond that, you must be attentive to any negative consequences. Perhaps the pushiness causes users to convert to a lesser extent on the organisation's most valuable offers.
• Find pages with low CTR in search engines and modify the page titles. It can also be worth modifying other metadata on the page. To find pages' CTR at Google, for example, you can check out their tool Search Console.

Image 57: Overview of my employer's CTR at Google.

Pogo-sticking

The user behaviour of jumping back and forth between pages applies mainly to pages that have a search-like interface, product listings or similar. It is about tracking how long a user stays on a subpage they reached via a list with many other alternatives. Whether this is a behaviour worth tracking on your website only you know. It is said that Google is curious about this to be able to compare different search results with each other. It is admittedly only one signal among several others, but there is surely reason to believe that a page's relevance is low if it is common for users to quickly back out.

Take two pages with identical relevance for a search query at Google. If one of the pages to a greater extent than the other causes users to come back, there is probably reason to lower its relative value. In the same way, you can think about how you yourself list content on your website. Is it worth keeping everything in the list, or can the list as a whole perform better if you make a change? What should be at the top and why?

Ways to try to reduce pogo-sticking:

• Improve the load time and see if it changes anything. It could be that you have a too-heavy image, or are waiting for a slow external API.
• Clearer confirmation of the page's value. It could be due to design factors where users do not understand or cannot decode the page's content and therefore back out. One approach is to try to reduce the cognitive burden for the user to figure out that they have ended up in the right place.

Dwell time

You need to be careful with this one, but at large scale it can be meaningful for knowing, for example, whether longer material is actually used by users reading it or dwelling on it. In other words, you need to cross-reference with a measure of how long it takes to absorb content. If it is primarily text, then the total reading time is a value to compare against. A function that can be integrated into your website. Content that has a high reading time and high dwell time gives a signal that the material is of high quality, has highly engaging content, or similar.

Now you might be thinking this is vain? Good! Dwell time is one signal among many others for relativising which content you in quantitative terms have reason to believe is appreciated by users.

Image 58: Estimated reading time for a blog post on my WordPress blog.

When I do a major content audit next time, this is an approach I intend to use. The alternative of doing it manually will guaranteed lead to scrapping appreciated niche content that is not heavily used but valued by those who found their way there.

It is not as complicated as you might think to combine the information you need. Your web statistics tool probably already has data on each page's average visit time. What you need to do is get the website to list the reading time for the addresses that exist. Then you merge the two sources and use the page's address as the common reference. I, who am more comfortable with databases, would have done the merge that way. But some people I know have not yet needed to leave Excel, since “everything you need is in Excel!”

From a search engine's perspective, or your product listing's perspective, dwell time can rather be seen as a measure of how long the user dwells on the search or product result's page before the user returns to the listing.

Net Promoter Score – NPS

You have probably encountered Net Promoter Score (NPS)⁵⁶ but perhaps not realised it. What you are trying to achieve with NPS is to have a single metric that should reflect how good something is, how loyal the customer or user is. The score ranges from –100 to +100. If the value is negative, it is a person or group who will more or less passionately speak ill of something. On the positive end is how much the person or group are ambassadors for something and talk about it in positive terms to those around them. Getting the score –100 means everyone is passionate in their dislike; getting +100 means everyone is so satisfied they want to tell those around them about it. Having above zero is considered good and above +50 is an excellent score.

NPS involves asking a single question, namely:

“How likely is it that you would recommend the company/product/service/website to a friend or colleague?”

The answer is given by the respondent on a scale from zero to ten, where to all nerds' delight you can actually choose zero. This result is then converted to the previously mentioned scale. Respondents are divided into three groups so you know what you are dealing with. Those who answered:

• 9–10 are called promoters and are ambassadors filled with positivity.
• 7–8 are called passives and do not contribute much benefit.
• 0–6 are called detractors, and you can assume they will not speak well of what they were asked about.

Answers of 9–10, promoters, are assumed to create added value in their personal network, for example wanting to tip off those around them when the opportunity arises. If you have met someone who has chosen to get an Apple computer, you probably know what I mean; the person seems convinced that the solution to almost all challenges exists within the Apple platform.

Those who answer 7–8 are considered passive; their behaviour lies between positive and negative.

Answers between 0 and 6 are considered negative (detractors). They will not speak well of your website when it comes up; they might even actively work against your business goals.

To calculate your NPS score, you count the groups' percentages against each other; for example, if you have: 35% promoters, 40% passives and 25% negatives, you follow the formula promoters minus negatives, 35 – 25. That gives +10 in NPS.

Remember to only compare your website's NPS score with comparable things. For example, you perhaps cannot compare a luxury car manufacturer's website score with the score for the Spanish embassy's website.

Image 59: Soundcloud follows NPS in their app.

SERVQUAL and RATER

Another model for letting users assess some form of service is SERVQUAL⁵⁷. It was developed in the late 1970s and is intended to be used to discover what gap exists between expected quality and that experienced by users. As previously mentioned, among other things with reference to David Maister, a formula for customer satisfaction is to compare the actual experience against the expectation that existed beforehand. If the experience is in line with the expectation, all is well, but if the expectation is higher than the experience, there will be problems.

To break down the perceived service of a service, which does not necessarily have to be digital, into a service's different dimensions, SERVQUAL has specified something called RATER. It is an abbreviation of the initial letters:

1. Reliability: the ability to perform the promised service reliably and accurately.
2. Assurance: the knowledge and courtesy of the service (personnel) users come into contact with. Also the tone in written text counts, of course. It is about users being able to rely on what is said and have trust in the service.
3. Tangibles: more applicable in a physical scenario possibly, concerns the tangible things like premises, inventory. In more digital contexts, possibly that the design confirms you have ended up in the right place.
4. Empathy: that you take care of the user's individual needs.
5. Responsiveness: providing service promptly enough.

Working according to this model is a mix of both qualitative input and a quantitative way to describe a current state. The qualitative aspect is that it is based on real users and their way of describing the service, but at the same time you can summarise each point with a score to try to quantify it. The advantage of quantifying it is that numbers are easier to compare over time, so you can know if the experience of the service has changed.

Triple bottom line

One way to categorise (and nuance) the goals you have is to weave in a bit of the organisation's social responsibility in your way of measuring. The concept was coined by John Elkington in 1994⁵⁸. In English, a concept you have surely heard of is used, CSR (Corporate Social Responsibility). You thus get a bit of morality into the work of follow-up.

The whole thing involves complementing the financial goals with social impact and what happens to the environment. In other words, you should balance a fair social climate, that the environment does not suffer, with also making money. Long-term thinking, quite simply.

Image 60: Video about the robot Liam that disassembles iPhones. Apple's values as part of their marketing communication.

Yes, it is hard enough to find wise and measurable business goals just around the financial. Being able to then relate and relativise the financial with social and environmental impact can feel overwhelming. At the same time, in a larger context it can be worth it. See, for example, how Apple's customer communication has got at least the environmental perspective incorporated. Everything from what environmental impact different products have, to opening a press conference by showing a robot that recycles old iPhones. The robot even has a name, Liam.

Because in a world with limited resources – some things cannot be replaced.
- Apple in their presentation⁵⁹ of Liam, their recycling robot

For me as an employee in a tax-funded operation, this is more natural now than when I worked in the private sector as a consultant. If our small Gothenburg office made a sufficiently poor financial result, you could count on it being shut down. If we instead made an excellent financial result, you would really be a killjoy if you thought we flew too much and took the train too little to earn that money.

It is not compatible with sustainable business to destroy the environment, or to hide revenues in a tax haven if it comes at the expense of the society you operate in.

Triple bottom line is a way of not getting tunnel vision about the financial, but rather realising that you have stakeholders who are not among the shareholders. Something I hope more and more companies start with as part of their sustainability work. Perhaps we web analysts can help this development along a little by choosing measurable goals in slightly unconventional ways.

About statistics (and common mistakes)

The data that does not have a clear connection to the business's most lofty purpose with the website is to be regarded as other data. They are not necessarily unimportant because of this but they are at least not obviously how you recoup the investment in the form of new customers, more orders placed in the shop, and so on.

Segmentation

Just as with other statistics, you can try to reshape your data in the hunt for findings, or because you want to see how the campaign to a certain customer group performs. A common trick, we have already touched on in the book, is to segment. Then you can focus on a subset of all statistics, for example checking whether the advertising mailout in Eslöv has a demonstrable effect on visitors from Eslöv on the website.

A common segmentation many web analysts did around 2008 with their hearts in their throats was to segment out mobile users. This to try to get to know their behaviour in the hope of not losing market share. It seemed inevitable that the majority would prefer to use their mobile to visit the web.

To understand the numbers being segmented, the segment may need to be compared with another segment. An interesting development we saw on 1177.se during the early 2010s was that the increase in the number of visitors could almost entirely be attributed to the segment of users with mobile and tablet. The development within that segment was explosive compared to those on desktop. Those responsible were fired up to get a responsive website that was a bit more user-friendly for those with smaller screens. The development was admittedly something most of us found hard to miss as it constantly came up. The question is just when, based on your own web analytics, you would have made an early decision to go responsive? The data we have on our own websites is what we have earned. That is, before the website works okay for a mobile user, there is a risk that that segment is so small that you do not discover it – since they do not come back more often than necessary.

Image 61: The development for the segment “mobile & tablet” accounted for the lion's share of the increase on 1177.se

The same question can be had about whether to optimise for users on PlayStation, or the few during 2016 who connect with suspect user agents⁶⁰ like “in-car”. How many connecting via their car do you need to see in your statistics before you should start caring?

Being aware of your limitations and difficulties

This business of interpreting data is difficult; that should not be swept under the carpet, and many times you should be sceptical about whether you have enough evidence to draw any meaningful conclusions. It is wise to be humble before your data, and if you have the opportunity to double-check your data and how you report conclusions with a statistician, that is time excellently spent. Regardless, make sure to have the explanation of method and report close at hand as you can count on questions being asked about the background to the conclusions.

A common problem specifically around web analytics is that you cannot draw far-reaching conclusions from collected data. Often because the change is not large enough to become statistically verified, or because the sample is too small (which makes the margin of error far too large).

When you as an amateur in statistics look at data, it does not hurt to remind yourself that you are human. People make mistakes, often. Furthermore, we make mistakes in sometimes predictable ways, which means you can have a list of common mistakes in your short-term memory before you cheer too loudly about something you have found.

Some suggestions for us web analysts are to be attentive to things like:

1. That the sample is okay and large enough. It is not statistics to talk about anecdotes from a handful of users' experiences; then it is about a qualitative method and that is something entirely different. Collected data needs to be from a sufficiently large study and furthermore the composition within that group is important. The sample should be representative, which is often described as needing to be sufficiently random.
2. Do not compare apples and pears. Both are admittedly fruits, but you must be careful with your conclusions. If you do an A/B test where one group consists solely of new visitors and the other of returning ones, it is risky to draw any conclusions.
3. Confirmation bias is the highly natural behaviour of looking for arguments and evidence that you are right. Meaning you place greater weight on things that confirm the view you already have. This does not need to be a particularly conscious action; rather, for many of us it can happen automatically – so feel free to tone down the accusatory tone when you point this out to someone.
4. Self-selection bias: That we ourselves are often not very representative. It is questionable at best to believe that you yourself belong to a group of users. Despite this, it is common to base things on yourself and form a user group based on that. Here you have thus ruined your sample.
5. Be careful with your causal relationships. We humans are very quick to draw conclusions on incorrect grounds; there is apparently an evolutionary reason for this. The downsides of this are superstition and seeing patterns where there is no connection. If you want something light-hearted about this, buy the book Spurious Correlations⁶¹, or if you want something more academic, the book Why: A Guide to Finding and Using Causes⁶² is what you are looking for.
6. Your conclusions never get better than your data. It is of the utmost importance that you have a handle on your data quality. That this data is representative and that nothing important is missing for daring to draw conclusions. Do you remember my example where users got stuck in the firewall? Before I know how common that is and how it works, it is hard to draw conclusions on the other data I have collected.

Try to operate within your limitations. When it feels complicated, read up properly or get in touch with someone who is an expert in the area. Admit when you do not know something, be open about it, and remember that not everything is actually worth finding out either.

Vanity metrics

In your web statistics you have masses of different views of data, at least more than you know what to do with. If you cannot tie them to something unambiguously good, they risk being interesting only because they appeal to your vain side. Now these figures are not completely worthless either, but it is important to see them for what they are. Namely figures you can only use as anecdotes when you meet people in your own profession.

Formulating goals based on the number of page views per visitor is questionable at best; you cannot really claim it relates to the goals the business has. Is it really always better to have many page views? Probably not, at least not always.

What you are still allowed to use with your honour intact from vanity metrics are those that tell the visitor's story about their session on the website. If you can tie the figures to a key performance indicator, or other valuable thing to be reported, then they are figures worth nurturing. Then they have a value, albeit a modest one. They can at best explain why a key performance indicator's trend break occurred. For example, that plummeting sales coincide with visitors from one month to another starting to use mobiles, or whatever the vain metric happens to be.

Hygiene factors

Hygiene factors are the prerequisites for your business goals to be able to occur. Why something should be able to occur. It is not the hygiene factors that make you money on your website, so they are not goals in themselves, but they are still important to monitor. They are the very foundation, or prerequisites for a value-creating website. These describe the craftsmanship that constitutes a well-functioning and appreciated website. What makes it possible to sell something, or for someone to engage.

The list of hygiene factors for a website is very long. Everything that is measurable, and that affects the user's experience, is part of what a web analyst has to worry about when looking for optimisation potential.

The last part of this book does a deep dive into a number of hygiene factors and other activities that can be part of systematic web analytics work, when looking for untapped potential.

Continue reading – Part 4: Activity plan