Results of KMail Icon Test #1
December 22nd, 2009 by Björn BalazsIt is great to see how many people participated in the first icon test for KMail: 3327 started the test and 2399 finished. This is a drop-out of about 28% across the whole study and fairly good for this kind of study - especially seeing the technical difficulties we had when we started the study.
Here is a first brief summary of the results:
- Selected language: English: 2244, German: 659, Spanish: 313, Polish: 109
- Gender: Male: 98%, Female: 2%
- Average Age: 29 years (from 10 to 88)
It is great to see that we got sufficient participants for reliable results in all languages!
As you might know, our test combines multiple indicators and calculates a single value for each icon-term relationship. The maximum value an icon-term-relation can reach is 10,0. Following I split up the results into 3 groups:
1. Icons working really well
All of the following icon-term-relationships show average values above 8 with no individual value below 7 for any language. We can assume all these icons to work well enough in all tested languages.
| Icon | Rating | Term |
|---|---|---|
 |
10,0 | |
 |
10,0 | Previous |
 |
10,0 | Trash |
 |
9,6 | Open Full Search |
 |
9,2 | Next |
 |
9,2 | Open a new tab |
 |
9 | Reply |
 |
8,8 | Create To-do |
 |
8,2 | Change Sort Order |
 |
8,2 | Select View Appearance (Theme) |
 |
8,2 | New message |
2. Terms with no working Icons
For the following terms no fitting icons were identified. They definitely need to be redone.
| Best Icon | Rating | Term |
|---|---|---|
 |
4,4 | Select aggregation mode |
|
4,2 | Templates |
 |
4,0 | Sent-mail |
|
3,2 | Filter by Status |
3. Icons that need some attention
For the following terms the icons show to be the best, but only in comparison to what was presented. They could not convince. The results here need to be further examined and icons should be discussed and - if needed - improved.
| Best Icon | Rating | Term |
|---|---|---|
|
7,8 | Outbox |
 |
6,8 | Forward |
 |
6,2 | Inbox |
 |
6,2 | Drafts |
|
5,6 | Check mail |
Note that for Inbox and Check mail the same icon won. There is the same problem with Sent-mail and Outbox. These concepts are obviously not correctly differentiated by the users.
Some cultural notes
With these studies we also want to learn about cultural differences and their impact on the quality of icons. So here are some interesting facts found in the results - perhaps some native speakers can help to understand why…
While this icon was well understood in most languages Spanish (’Responder’) and Polish (’Odpowiedź’) speakers had much bigger troubles with this icon.
Only German speakers (’Weiterleiten’) understood this icon really well - all other had much bigger difficulties.
Only Spanish speakers (’borrador’) understood this icon really well.
Spanish speakers (’Crear tarea pendiente’) had much more problems with this icon.
Should anyone need the results in greater detail, please let me know. Under certain circumstances, I can make it possible to directly access the results. I planned to start the next test before christmas, but I won’t manage to do so. But we will definitely have Italian available then, and perhaps also Norwegian… Wishing all of you a happy christmas!
Tags: cultural impact, icon test, kde, Results
December 22nd, 2009 at 20:57
I, as an English-speaking Swede, can comment on the Forward icon. When I did the test, it was the word “forward” rather than the icon that confused me. If I had thought of “forward” as in “to forward”, instead of a synonym to “next”, I would without doubt selected the “right” icon.
December 22nd, 2009 at 21:14
@Sebastian Gröhn / About ‘forward’:
That is a good point. That icon-term relationship does not work out, because non-german speaking people chose the icon with an arrow to the right (icon for ‘next’) quite often. Consequently the german result for ‘next’ is much better than the results in other languages. And in German “Weiterleiten” is without any doubt an action, which is associated with “to forward”.
Given the context of KMail this will probably mean that the icon-term-relationsship is good enough for keeping this icon. Or is there a better, less ambiguous term than ‘forward’? One that direectly implies an action?
December 22nd, 2009 at 21:35
forwarding?
December 22nd, 2009 at 21:43
Regarding Forward, I think this is simply the lack on context noted by several people in a previous blog entry. Only presenting “Forward” lacks the required context this icon is used in. I’d guess the test would yield even better results, if you’d present not only a single term like “Forward”, but instead a short description like “Forward this mail”. This is probably the intention the user has in mind when looking for an appropriate icon. I had a similar problem with one of the folder icons (don’t remember which one), where an additional “Ordner” in the description would have lead me to the right icon.
Regarding “Select aggregation mode”, this probably not only fails because the icon is suboptimal, but also because people don’t have a clue what the term means. Not only in you test, but also in KMail itself. What I want to say is, you can come up with the perfect icon, still a lot of people will not assign it correctly. At least I only thought “Huh?!” when I read it…
December 22nd, 2009 at 21:59
I have these alternative suggestions for the not working icons:
“Filter by Status” - A sieve, e.g. a sieve used for composting, this would be also different from the otherwise used omnipresent coffee filter (which by the way often dos not get recognized as such)…
“Sent-mail” - A nice post box (public post box for collecting letters, not the private one ;-). When you send mail you exactly do hand over your mal to the service provider that cares about sending it to the addressee.
“Templates” - Some toy blocks. Creating a template can be seen as combining single elements to a flexible building.
“Select aggregation mode” changes the display of mail headers (sender, topic…). So I would suggest a front view of a full book case or if that is too complicated several backs of books or folders in a row, as that’s usually that what you see when you poke through a library…
What do you think?
December 22nd, 2009 at 22:46
Well, with the forward and reply icons I personally had the trouble with the different colour on the icons. Why is it different? For these actions feel the same, so only the directional arrows should change…
December 22nd, 2009 at 23:09
wow the results were better than I expected.
Thanks Björn, and every one taking the test.
#note Filter by Status is what ever filter is selected
So there is no real icon for that action.
This test does not reflect another aspect that is context, some of the icons in this test are in a certain context, that produces subgroups of icons…(like folder like icons) and status icons of the mail.
cheers nuno
December 23rd, 2009 at 6:16
@Arnomane
I think a post box for sent-mail is problematic, because post boxes tend to look completely different in different countries (examples: yellow cubes in Germany; green pillars in China)
December 23rd, 2009 at 10:57
IMO, you can’t conclude on a cultural aspect because selected language isn’t necessary native language. For my self, I selected English but I’m French… I don’t understand how you can conclude as quickly with cultural aspects…
December 23rd, 2009 at 11:28
in Polish ‘reply’
when referring to noun it should be translated:
‘Odpowiedź’
when referring to verb in imperative form it should be translated:
‘Odpowiedz’
(the last letter is different)
the second form is more common e.g. outlook is using it.
I’m not sure how it is translated in kmail. Maybe only it was not translated correctly in this usability test. I’ll check at home.
December 23rd, 2009 at 11:58
Hello!
Very interesting results. I’m very interested in the age distribution of Kmail’s users. Could you publish a histogram and/or median and standard deviation?
Thanks
Jochen
December 23rd, 2009 at 13:53
@Patrick: I am aware of that problem but you can avoid it very easy: All post boxes have a quite similar letter slot. So an icon of a letter half inside of the slit of a letter box (no matter which color) is a perfect icon for “sent-mail” in any language and any country.
December 23rd, 2009 at 14:02
@nuno pinheiro: “Filter by Status”. Well any filtering or searching does not exactly tell you what you search or filter for.
So I think a sieve (which can filter a lot of different things, such as sand, gravel, soil, …) would fit quite good.
December 23rd, 2009 at 14:24
@nuno pinheiro: thanks for your tip. We will test those icons used in there as well.
@Jochen S.: Will publish those after christmas
December 23rd, 2009 at 17:19
Björn, this is great work but I’m worried about the statistical strength of the conclusions given the number of participants, the number of tests and the division by languages of the participants.
Have you considered the statistical strength of your conclusions? Did you
(1) set out the statistical tests you would make at the start
(2) calculate the strength of those tests after getting the data
(3) consider the tests as a whole (the net probability of getting an anomalous result from the many tests)
The results are obviously useful in any case, but some of the results might benefit from follow-up.
December 24th, 2009 at 15:38
The reason why I selected the same icon for “Inbox” and “Check Mail” (the one that you show here) is that I associate the icon to both the action of checking mail and the result of that action (the mails are in the inbox), because of that huge arrow on it (this suggest the action of downloading the mails) and because of the box (this suggests the place where the downloaded mails arrive). Similarly, I chose the same action for “Outbox” (messages waiting to be sent) and “Sent Mail” (the result after sending).
I believe that the three icons “Change sort order”, “Select aggregation mode” and “Select view appearance (theme)” add to much clutter to the interface (furthermore I cannot configure them away as in the usual toolbars). These items should be in the “Settings” menu. Who is pressing these buttons regularly? I only pressed them the first time that I saw them and after the initial configuration I never pressed them again. If they are in the menus, then one can still add them to the regular toolbar if they are needed.
The icon test could be improved with the following addition: display the icons to the user and ask them what they mean (actually this is what users have to do when they use the program: they see icons and they have to guess their meaning and click on the right icon). For example, when I had to choose an icon for “Select view appearance (theme)”, I selected the correct one because no other icon fitted that description, but if the icon were presented and I was asked what it means, I would have never guessed the correct interpretation. In KMail, each time I look at that icon I have to hover the mouse over it to see its description (and similarly for its two neighbours). Unfortunately I am unable to suggest something better for these three icons (I would probably even not have been able to come up with an icon that good).
I think that the cultural differences may also be the result of chance: for example it was pure luck that I selected (in the English test) the correct icon for “Create to-do” (I chose it because no other icon fitted), but I don’t think that the meaning of the icon is obvious. The check-mark in the icon doesn’t have a clear connection to the notion of “To-do” in my opinion. It would be clearer if the icon showed instead of the check-mark a list of items (two or three) with a check-mark before all items except the last one (representing the to-do item that you are adding). I am however not sure if my idea scales well to smaller sizes. Maybe an icon similar to the one in the Buuf or the Buuf Deuce icon set (found on kde-look.org) is better.
Maybe “Forward” could be renamed as “Forward to”.
December 26th, 2009 at 20:17
@maninalift:
We have done such statistical calculations with data of systematical test studies. We manipulated the degree of difference within the iconset. The test proved to produce reliable results with about 30 participants in a random online-sample. Less participants are sufficient when the sample is hand-selected. So we can assume that the results in all languages are valid.
I am willing to support you (and anyone else), if you are interested in calculating statistical values and thereby are willing to help to improve the test. Just contact me.
December 26th, 2009 at 20:24
Thanks for all the comments on why icons were selected and how icons could be improved. I am sure we will use all these comments when actually discussing how to improve which icons.
@g I just took a look at the statistics of “Select View Appearance (Theme)” - you are absolutely right: this icon was chosen as best of the non-fiiting icons. I overlooked this in my first review. I would now say we should think about this icon as well.
January 4th, 2010 at 16:28
[...] los resultados de la primera encuesta de KMail en el bolsillo, el equipos de usabilidad de Kmail inicia otra encuesta con el fin de mejorar la [...]