OpenSource Usability Labs

Today I closed the 2nd KMail icon test. This time 1945 people took part. Much less people than the last time, but also the drop-out was much less than last time.

So here are some basic statistics:

• Selected language: English: 1002, German: 476, Spanish: 334, Italian: 76, Polish: 57
• Gender: Male: 99%, Female: 1%
• Average Age: 28,6 years (from 11 to 93)
• Most of the participants are strong KDE users. More than 80% use KDE very frequently. Next is Windows (TM) with more than 25% of frequent users, GNome with about 10% and Mac OS (TM) with about 7%.
• The participants are almost half split in their use of KMail, more than 50% use KMail frequently and about 30% say they never use KMail (the rest in somewhere in between).

In this test we included the icons of two elements of KMail: The icons used in the window and in the message menu of the composer window. These are icons of different size in the standard configuration of KDE, so we presented the icons in the according size and used the KUbuntu translations of the terms associated with them. The overall findings are very good. Most icon-term-relations seem to work out fine. We did not find find any explicitly not working relations, but some still show the potential for improvement.

Icon-Term relations working really well

The following icon-term relations seem to work well. (more…)

It is great to see how many people participated in the first icon test for KMail: 3327 started the test and 2399 finished. This is a drop-out of about 28% across the whole study and fairly good for this kind of study - especially seeing the technical difficulties we had when we started the study.

Here is a first brief summary of the results:

• Selected language: English: 2244, German: 659, Spanish: 313, Polish: 109
• Gender: Male: 98%, Female: 2%
• Average Age: 29 years (from 10 to 88)

It is great to see that we got sufficient participants for reliable results in all languages!

As you might know, our test combines multiple indicators and calculates a single value for each icon-term relationship. The maximum value an icon-term-relation can reach is 10,0. Following I split up the results into 3 groups: (more…)

To the nearly 200 (173, to be exact) participants in our icon test: Thank you all for taking part! We got great results in terms of the quality of the Tine 2.0 icons (which are, in fact from the Oxygen Iconset). Also, we would like to thank you for all the comments you gave: Many of them were really encouraging, we got many compliments, but also some valuable criticism and tipps.

How did the icon test work?

We not only recorded which icons were chosen for which terms, but also how long it took you to decide.

From that, we could calculate three values:

• “Strength of association” (Indicates what percentage of the users assigned this icon with this term.),
• “Discriminatory power” (Indicates wether and how often this icon was assigned to other terms as well.) and
• “Conspicuity” (Indicates how fast the icon was chosen by the user in proportion to the average.)

In the pictures below you can find the three values on the left in the above order. The three values together give an overall Rating between 0 and 10. After a few tests we conducted, we can say that only icons with a rating above 8 can be seen as perfect for a term.

What is very apparent in all the results: The Oxygen Icons are not made for Tine! From the few terms we wanted to find the perfect icon for, only “User” got a rating above 8:

The weakest rating was achieved for “Today”, which is very interesting, because I expected this to be the most evident one:

Here are the results of all the other terms:

Add Appointment:

Add Calendar:

And Resource:

What do we learn from that?

First of all, we need icons that fit better to the special Tine 2.0 terms. Of course, the Oxygen Icons are a great start, but they just don’t fit all of our needs.

That’s why we want you, our community, to contribute!

• If you use Tine 2.0 and you see an icon which you think doesn’t fit: Tell us!
• If you are an icon designer and love to play with pixels: We need you!
• If you know how to visualize complicated things with as little complexity as possible: Write us!
• And even if you just know someone of the above (or know someone that knows someone :), we would like to hear from you!

Please write your ideas and suggestions to: tine.calendar@gmail.com or post in the Tine forum.

Thank you for taking the time answer our survey!
We had a great outcome with a total of 365 successfully completed surveys and 106 partially finished ones.

A few things that came out that you might find interesting:

• Overall, 346 people took the survey in German, 68 in English
• 30% were women, 70% were men
• on average, people were 31 years old
• 85% come from Germany, 10% from the Rest of Europe, the remaining 5% are from other parts of the world including China and South Africa (!)
• Professional appointments: women mostly use papercalendars (65%) and locally installed calendar applications (44%), men mostly use locally installed calendar applications (54%) and Smartphone calendars (46%)
• Private appointments: women mostly use paper calendars (83%) and locally installed calendar applications (24%), men mostly use Smartphone calendars (49%) and locally installed calendar applications (45%)
• The calendar application used the most is Microsoft Outlook (45%) followed by others (25%) and eGroupware (23%)
• 43% of all people asked use a smartphone, the biggest smartphone brand being Nokia (30%) followed by Apple iPhone/iPod touch (27%)
• 1/3 of smartphone users synchronize daily between phone and computer
• Per week, 3/4 of all respondents make one or no all day appointment, 2-10 appointments shorter than one day, none with international partners and 1-5 recurring appointments
• Half of all people who answered don’t make any multi-day appointments per week
• Nearly half of all people (45%) don’t make any public appointments, while 2/3 of them make 2-5 private appointments per week
• Nearly two thirds (65%) of the people asked find it very important or important to include tasks and milestones into their calendar
• Slightly more than two thirds of all respondents find it very important or important to make appointments distinguishable by colour
• But there no noticeable tendency when it comes to accessing other calendars to coordinate appointments: 39% say, that this is absolutely not important or not important to them, while 34% say the opposite. 20% are inbetween, though.
• An clear majority of everyone (86%) who answered wants to have a “quick save possibility”, with start (98%), name (92%), place (67%) and end (58%) being the most important information for creating an appointment.
• As for the calendar view, people considered the week view to be the most important one followed by the day view.
• Also, Pop-Up seems to be the most favourite way of all the respondents to be reminded of an appointment, 2/3 (67%) voted for that, followed by sound signal (48%).
• And for the last question: As you can see in the picture below, synchronization is what people miss most in their current calendar, followed by reminders and groupware functionality:

If you have any questions regarding the survey, please ask!

The Icon Usability Test of the TV-Browser icons has been closed for some time now. We have been busy learning from that study and working on further improvements for the icon test. So I totally forgot to write about the actual results…

Just as a reminder - the aim of the study was twofold:

First, we wanted to prove the method, which worked very well and we are going to publish the results in some scientific magazine sometime.

The second goal was to reveal the good and the not so good icons in TV-Browser. For the test we used some icons that are visible in TV-Browser and the tool-tips provided for them. We then asked the user of TV-Browser (they should be used to the icons!) to attend the survey - and they did indeed! A great “Thank you” to all of you who have participated!

And these are the results in short:

The following icons proved to be just great and really helpful for the user:

Print, Reminder Function, Favorite program, Search, Filter

A second set of icons worked alright, but has the potential to be improved:

List of currently running programs, Simple marker plug in

And finally some icons showed the definite necessity for an improvement:

Start external Program, Tool for translators, Get recent TV listings, Rate program

Now we are working on an second study, in which we will test alternative icons for those last functions. In this study we are going to present alterative icons to the users, which then have to pick the icons that fit best for he function. At last we are going to test the new set of icons to prove the improvement, that we hopefully will have achieved by then!

Next to TV-Browser, we are also working on a study for the testing of Tine 2.0 icons. Hopefully I will be back to you with some news on that issues soon!