Usability, Experience, and Evaluation Study

Related usability studies: March 2009, September 2009

Methodology

In March 2010, the Wikimedia Foundation Stanton Usability Team partnered with gotomedia to conduct a third and final evaluative study of the Wikipedia editing experience. This consisted of ten one-on-one in-person interviews and eight remote interviews. Sessions began with a brief interview about participants’ computer use in general, and their past experience with Wikipedia. In this usability testing, participants from a representative sample of website users were observed as they performed a set of predetermined tasks from a script. The 75-minute sessions were led and moderated by an experienced Usability Specialist. Throughout the session, the UX Specialist asked additional questions to clarify and expand on participant responses. This process helped to create an understanding of how users interact with the website and illuminates potential usability problems.

The primary research goals were to:

evaluate the progress/improvements made by the stanton team in reducing the obstacles that novice users encounter in editing a Wikipedia article, including—but not limited to—adding personal content, fixing a typo, adding a reference, formatting content, and creating an article.
evaluate the new prototype interaction with templates
evaluate the effects of template collapsing.

Target audience

Participants were chosen based on the segmentation agreed on by gotomedia and Wikipedia, focusing on Wikipedia users who had limited or no editing experience.

Participant No.	Has not edited, but is willing to	Has not edited, doesn't see themselves editing	Has made < 25 contributions	Male	Female	Under 18	19–34	35–54	55+	Uses WP everyday	Uses WP > Once a week	Uses WP Once every few weeks
01	X	X	X	X	X	X	X	X	X	X	X	X
02	X	X	X	X	X	X	X	X	X	X	X	X
03	X	X	X	X	X	X	X	X	X	X	X	X
04	X	X	X	X	X	X	X	X	X	X	X	X
05	X	X	X	X	X	X	X	X	X	X	X	X
06	X	X	X	X	X	X	X	X	X	X	X	X
07	X	X	X	X	X	X	X	X	X	X	X	X
08	X	X	X	X	X	X	X	X	X	X	X	X
09	X	X	X	X	X	X	X	X	X	X	X	X
10	X	X	X	X	X	X	X	X	X	X	X	X

Recruiting

As in past studies, we recruited over 2,000 San Francisco residents for the in person testing and over 1,000 US residents for the remote testing. A banner/alert was placed and displayed at the top of every 1 in 100 Wikipedia pages for the duration of 3 days. When users viewed our message by clicking, they were forwarded to ethnio recruiting system where we asked them a series of questions. Specifically, users were asked about what they were doing on Wikipedia, how often they used Wikipedia, if they had ever contributed to Wikipedia (and in what manner), their age, gender, location, and availability. Based on these criteria, the 3,000+ users who responded to our survey were filtered down to 500 viable subjects based on their answers to these questions. The team and gotomedia worked with Fleischman Field Research to contact, filter, and schedule these participants based on a further phone conversation covering their Wikipedia usage patterns, reasons for not contributing, talkative and o

Study Participants

Remote Testing

Participant No.	Age	Gender	Wikipedia Editing History	Wikipedia Usage	Occupation
01 "Phil"	22	Male	Has not contributed, but is willing to.	Every day	Freelance writer
02 "Keith"	41	Male	Has not contributed, but is willing to.	Every day	Technician
03 "Bianca"	22	Female	Has not contributed, but is willing to.	Every day	Intern
04 "Adam"	21	Male	Has not contributed, but is willing to.	Every day	Student
05 "Charles"	46	Male	Yes. Fewer than 25 edits.	More than once a week	Court Clerk
06 "Nicholas"	31	Male	Has contributed. 1 article.	Every day	Account Manager
07 "Meghan"	19	Female	Has contributed once.	Every day	Homemaker
08 "Lisa"	52	Female	Has edited. ~10 contributions	Every day	Homemaker

Scripting

The moderator script was drafted by the Wikipedia Usability Team and gotomedia, based on the script originally developed and used last year. An almost identical script was used for the lab subjects and the remote subjects, the discrepancies being entirely related to the difference in set up. The script aimed to see how users would attempt to and feel about a collection of tasks or objectives that first time and early Wikipedians frequently encounter in their editing process including, but not limited to:

Finding and using the various modes of editing ("edit this page" vs. section edits)
Adding personal content or information to an existing article
Looking at Discussion Pages
Adding comment/thread to a discussion page.
Adding or using content from an external source (website, paper, etc)
Adding a reference to content
Adding an external link to a list or section of links
Editing an article that uses a template
Editing an infobox through wikisyntax and pop ups
Formatting content (Headers, bold, italics, linking, tables)
Creating a table and new article
Fixing a typo

Though the script was generally task based, the questions were open ended and left room for interpretation, encouraging subjects to not try to complete the tasks in any "correct" way, but rather to go about them naturally, if they had felt the motivation to do so on their own. Open-ended interview questions including "How did you find that process?" regularly follow every "task". The script was piloted on one staff member of gotomedia, a reader (but not editor) of Wikipedia prior to any official testing.

View the actual Moderator Script here.

Lab Interviews

Remote Interviews

Summary of Results (by gotomedia)

Overview

As a part of the Wikipedia Usability Initiative, the Wikimedia Foundation asked gotomedia to conduct a series of usability tests to evaluate progress toward the Initiative’s overall goal of “measurably increase(ing) the usability of Wikipedia for new contributors.” In previous rounds of testing, Wikipedia and Bolt | Peters uncovered and documented many usability issues that exist in Wikipedia generally, and in the editing process specifically. Our research reaffirmed these previously documented issues such as:

People love Wikipedia. Wikipedia is extremely popular with users. All of the participants we spoke to use the site at least several times a week, and many use it several times a day.
People don’t think they have anything to add. Nearly everyone we interviewed reported that they are reticent to add to Wikipedia because they are not confident in the quality or validity of their own potential contribution.
Wikitext is easy ...and hard. Users are able to make simple text changes to a Wikipedia article with relative ease, but more advanced editing tasks such as creating links, footnotes, and tables proved to be more challenging.
Users learn by example. When confronted with a task that they did not immediately understand, most users looked for examples in the Wikitext and attempted to mimic what other editors had done.
Users are not able to create a new article. Only one of our participants was able to figure out how to start a new article on Wikipedia.

These issues are well documented. We did not find anything in our research that significantly contradicts any of the findings of the previous studies. We will focus our comments on more specific editing issues, new functionality that was introduced since the previous tests were conducted, and additional observations not previously documented.

General Editing Issues

Wikitext presents significant barriers to editing. The best way to really open up the editing process to all users would be to hide Wikitext from users altogether. Wikipedia participation will only be truly open to everyone when a true WYSIWYG interface can be developed. There are many technical, cultural, and financial reasons why a true WYSIWYG interface probably won’t be implemented at Wikipedia any time soon, and there are legitimate arguments about whether or not making editing that easy would ultimately be good for Wikipedia. However, if the goal is to truly open up editing to all users, a true WYSIWYG interface would be the best way to accomplish that goal.

“I am not a technical person so I have no idea (how to do this).” -Carrie, 32, Account Manager

“...seems like that would be so much easier than dealing with code. The way it’s doing it now, it seems to be keeping some people out. It seems exclusionary.” – Phil, 22, Freelance Writer

“I know my mother couldn’t use this!” – Oliver, 48, Business Analyst

New editors need more help. Several users expressed a desire for a short video or tutorial that would give them a basic understanding of the editing process. Many users were observed looking for help in the left hand column of the page, and those that did so expected to find edit-specific information under the Toolbox menu in that left column.

“Some of the things aren’t actually intuitive. So either they could make it more obvious for a novice or lay people, or make a tutorial. (And) make that tutorial really, really obvious for when features are just rolled out.” – Bianca, 22, Intern

“I think they should highlight the Help and how to contribute. It should be super obvious on the front page. I had no idea and had never tried, but I think if it were more obvious (how easy editing is) then more people like myself would.” – Adam, 21, Student

“I would likely look at a video. I know Google does that for their suite of apps...” – Clayton, 22, Tutor

Editing Toolbar

The editing toolbar definitely represents an improvement for editors over the tools previously available to them, and the changes made to the toolbar since the previous rounds of testing do appear to facilitate easier editing. Toolbar visibility was improved over the previous design. Some participants still overlooked the toolbar, but most users did eventually find the toolbar on their own.

Internal links confused many users. In the first round of testing by Bolt | Peters, Internal Links and External Links were accomplished using two different tools in the Toolbar. In that scenario, users tended to find one of the link tools, and ignore the other. So in the second round of testing by Bolt | Peters, the two tools were collapsed into a single link tool, and users were expected to choose an Internal vs. External link by selecting a tab at the top of the overlay. But in this scenario, users tended to overlook the tabs. So in this third round of testing by gotomedia, the Internal and External tabs were eliminated, and the overlay would automatically determine which kind of link the user wanted to make, based on what they entered into the “Target page or URL:” field. This was an elegant solution, but still confused many users. Many participants assumed that they would need to enter the full URL of the Wikipedia page they wanted to link to. So they would open a new tab or window, navigate to the target Wikipedia page, copy the URL, and paste it into the Target page or URL field in the overlay. The overlay would then treat the link as external, inserting the full URL into the Wikitext, resulting in a working internal link that displayed as an external link. That is the functionality that in-person participants encountered, but this functionality was changed before the remote interviews were conducted. If remote participants entered a full URL at Wikipedia.org, they were shown a second overlay that said, “The URL you specified looks like it was intended as a link to another wiki page. Do you want to make it an internal link?” With this dialog implemented, participants did end up with the correct link format in their edits. The progression of the Link tool over the course of the usability initiative has grown more and more successful, but there is still some room for improvement.

The Reference tool proved very challenging. None of the participants recognized the tool as “Reference” unless they discovered the mouseover tooltip. Two participants mentioned that they would have expected that icon to mean “bookmark.” When users found the Reference tool and tried to use it to add a citation, they universally failed to accomplish the task correctly. If they highlighted text in the article before clicking the Reference tool, then they were surprised to find that the text “disappeared” from the article when they previewed or published their edits. Few users ever found the text where it had been relocated in the References section of the page.

When participants attempted to add a reference without using the Reference tool, they were similarly perplexed by the process. Many users expected to add their reference directly to the References section of the article. So they would use the inline “edit’ link for the References section, and then be completely lost when they found only the Template:Reflist tag there and none of the actual references they expected to see. Even when users found the Reference entry in Help and tried to mimic it, they still were not sure what information should go where, or how their references would appear on the page.

“That picture does not look like a reference picture to me at all.” – Rob, 30, Student

“If you clicked reference and a box showed up, like a popup, that let you put in the title and URL of your reference, that seems to me like it would be so much easier than trying to deal with code.” - Phil, 22, Freelance Writer

There seemed to be a general confusion about the expected function for the reference tool and the linking tools. For the latter, highlighting is proper. For the former, it is not. Yet, users tended to conflate these two processes together. While these two processes are not, strictly speaking, connected with each other, users do seem to see them as somewhat analogous when it comes to highlighting. Therefore, some exploration should be done to consider how these two processes can be better aligned.

The term “Advanced” scared some users away. The Advanced label intimidated some users. Those that were already feeling overwhelmed by the editing tasks said that they were already in over their heads, and did not want to add anything that was “advanced.” In previous testing, participants expressed that they were overwhelmed by the large number of tools in the toolbar. The Advanced section was implemented to hide the less-often used tools until they were needed. This was a good direction to take, but the labeling should be reconsidered.

“Advanced should be called something different like formatting because advanced scares people.” – Caroline, 25, Project Manager

Help is indeed helpful, but could be expanded. Users appreciated the presence of the Help ‘cheat sheet’ embedded in the editing interface. The embedded placement represents a significant improvement over previous designs in which users had to interrupt their editing tasks to go find the cheat sheet on their own. However, the cheat sheet does not include the new tools in the toolbar. Several users went to the Help section to investigate how to make a reference for example, but the cheat sheet does not direct them to use the Reference tool.

(Clayton was asked to add a Reference) “I am trying to find a help button. (He opens Help module.) I like this and I can look at it and edit at the same time. I like it a lot. ...I am trying to add this reference. I don’t know what “test” in quotations means. I guess this is the link outside, ...additional text...and reference. I would just try it.” (Then after struggling with the Wikitext for several minutes): ...Oh I guess I overlooked this. (He finds the Reference button.)” – Clayton, 22, Tutor

Participants did not understand the Embedded File tool. Participants were not asked to complete a task using the Embedded File tool, but several of them commented on the tool. Nearly all users who commented on the tool said that they expected the tool to insert a picture. But the mouseover tooltip “Embedded File” confused them. If they clicked the tool, then they found that the markup that was inserted included “.jpg”. This confirmed their first thought that the tool was for inserting a picture, but left them wondering why the tooltip didn’t simply say “Insert Picture.” The Insert Picture Gallery tool added to the confusion. Both icons were read as “Insert Picture”, but users did not understand why there would be two tools, and why the tooltips were different.

“This embedded file is not well-named. Should be embedded photo because I don’t know the difference between photos and files.” – Caroline, 25, Project Manager

Navigable Table of Contents

Participants universally liked the Navigable Table of Contents (NTOC). Not all participants noticed the NTOC without prompting, but once they did, they seemed to understand it quickly. Some did not experiment with clicking the NTOC. Those that did understood its function immediately.

“The table of contents is also very helpful because this is a very long article. The table of contents is helpful, a nice little map.” – Caroline, 25, Project Manager

“The right hand navigational part... It makes sense and it’s a lot more useful than what Wikipedia usually has.” – Bianca, 22, Intern

Participants were pleasantly surprised to see a new section appear in the NTOC immediately. Users were asked to create a new section in an article. Many users stumbled a bit on this task. Some assumed that they would need to edit the TOC directly, and so began looking for an inline ‘edit’ link for the TOC. Others looked for a tool to help them create a new section. But once participants began examining the Wikitext, most found the existing headings and were successfully able to mimic them. Once they did, nearly all of them were surprised to see the new section appear immediately in the NTOC. Participants found this immediate feedback to be very reassuring. This is perhaps the only example in the editing interface where users get instant confirmation that their edits are being made successfully.

“It’s nice that the headlines are automatic. That they automatically create a table of contents.” – Caroline, 25, Project Manager

Preview

The Preview state needs clear visual cues. Participants often confused the Preview state with the Read state. Users would make an edit in Wikitext, click Preview to confirm the edit, and then forget that they were in Edit mode. They would click links in the article and not understand why the links did not work, or they would click an inline ‘edit’ link to try to make another edit and find that the edit link simply did not respond. Particularly for new editors, this could lead to a feeling that they had broken the page or done something wrong.

“I didn’t like the fact that when I was on a Read page that it just didn’t look any different from the Edit page.” – Tram, 30, Student

Some users were reluctant to use Preview. Some participants went immediately to Publish rather than to Preview after making an edit in Wikitext. In most cases they simply did not notice the Preview option. But in some cases participants continued to use Publish rather than Preview, even after Preview was pointed out to them. This may have been due to the special circumstances of the testing setting. During testing, participants were encouraged not to worry too much about the accuracy of their edits, so that we could concentrate on the editing process. This may have led participants to be more comfortable with publishing than they would have been in a natural editing scenario.

“I usually just publish it and make changes after. I’m usually scared of losing my edits even in Word Press. It’s really easy to navigate out of page and it rarely saves itself. I usually publish first.” – Bianca, 22, Intern

“Preview: I didn’t see it at first, probably because I wasn’t looking for it. ...you automatically want to go with pushing the (Publish) button next. So maybe moving the Preview over here (near Publish on the right side).” – Caroline, 25, Project Manager

Link behavior in Previewmode needs more attention. Currently, most links do not work in Preview mode. Links to References do work when the full article is being edited. These same reference links do not work when a single section of an article is being edited. Study participants frequently tried to use links in Preview mode. One commonly observed scenario proceeded this way:

Use the Edit tab to display Wikitext for an entire page
Make an edit
Click the Preview tab
Scroll down to find and confirm the edit
Realize that the edit was incorrect or that another edit was needed.
Click the inline ‘edit’ link to correct the edit
Nothing happens.

The inline ‘edit’ link doesn’t work and the user does not understand why.

If the Preview pane is to remain within the primary editing window, then a consistent, predictable behavior must be developed to handle links within Preview mode. That behavior could be enabling all links, enabling certain kinds of links, or disabling all links. For any links that are disabled, users should receive clear consistent feedback about why the link does not work as expected. If a user clicks a link that is disabled, a dialog should instruct the user, “You are previewing this article. Links do not work in Preview mode. Return to the full article to enable links.“ Our recommendation is that at least some kinds of links should work in Preview, including newly-added links, links to references, and inline ‘edit’ links. Inline ‘edit’ links provide a more immediate way for the editor to correct their edits. If for some reason inline ‘edit’ links cannot be enabled in Preview, then they should be hidden in Preview mode.

Preview behavior should be consistent. The behavior of the Preview tab at the top of the page differed from the behavior of the Show Preview button at the bottom of the page. The Preview tab would replace the Wikitext pane with the Preview pane, while the Show Preview button would display the Preview pane above the Wikitext pane. If a user used one method to access Preview mode and then used the other, they were frequently confused by this inconsistent behavior.

Publish

Users do not understand the meaning of “Minor edit.” During the publishing process, editors are presented with the option to check a box indicating that their edit is a “Minor edit”. Nearly all participants checked this box, assuming that because the changes that they had made were relatively small, that they would be classified as “minor.” In fact, Wikipedia would have classified the majority of edits made during the study as ‘major’.

“I’ve never figured out what minor edit means. I usually click on it since all my edits are small so I would assume that’s what it is.” – Bianca, 22, Intern

Links in the Publish overlay are not recognizable. When users click the Publish button, this overlay displays: Many participants mentioned that they did not understand all of the items mentioned in this overlay, particularly the license information. But none of our participants discovered that this overlay contains links that lead to more information. None of the links on this overlay display as such.

Infobox Editing

Several users expressed an affinity for the infobox. They look to the infobox for relevant statistics and “hard information.” They appreciate the immediacy of the infobox and the way that it “surfaces” important information for easy access. When participants were asked to edit information in the infobox, many expressed surprise or even doubt that the information there was editable. Some believed that the information in infoboxes was automatically populated from external sources, and/or that there would be additional barriers to editing this content. Users expressed more confidence in the accuracy of content in the infobox, as compared to content in the main article.

“In the past couple of years they have added this right hand window with key facts and I love it. It’s very easy. It’s easy to get all the key info on the right.” – Caroline, 25, Project Manager

Users had a hard time figuring out how to edit the Infobox. When prompted to edit information in the infobox, most users looked for a corresponding inline ‘edit’ link. They often clicked an inline ‘edit’ link that happened to display near the infobox. When that did not work, they then used the Edit tab to display Wikitext for the full page, and looked for something that would correspond to the infobox in the NTOC. Very few participants recognized the infobox settlement template capsule as the place to edit the infobox without prompting. (More on the template capsule below). Participants also struggled to name the infobox. When looking for a way to edit it, they didn’t know what the infobox was called, and so didn’t know exactly what they were looking for. “The infobox could be named something else. I don’t know, but I can’t think of another clever name. But it’s not obvious that’s what this is. I’d say “the side bar” maybe. ...I don’t know.” – Caroline, 25, Project Manger

“Well there are lots of these little things (templates). So it just looked like everything else here. Its because of the name. We don’t know what to call it.” – Oliver, 48, Business Analyst

Template Capsules

Templates were introduced to the editing interface in an effort to further hide markup, and increase the readability of Wikitext. They are largely successful in that effort, but the design and implementation of the template capsules need refinement to improve understanding of their use and function.

Some users couldn’t see past the triangle. Template capsules offered two primary means of editing the markup they represent. Editors could expand the capsule to show the full markup by clicking the triangle icon on the left of the capsule, or they could click the capsule title to open up an editing wizard. About half of our participants clicked the arrow first, and assumed that was the only way they could interact with the capsule. They had to be prompted to click elsewhere on the capsule to discover the wizard. The opposite was also true. There was, however, no correlation between the function discovered and the user’s comfortableness with editing Wikitext directly. Therefore, a code-anxious user may click the expand arrow, and be stuck, or a code-friendly user might pop the overlay, and be frustrated. So, rather than providing options to two types of users, the UI has the potential of confounding both while serving neither.

Given that the test participants were inexperienced editors, we found that once these participants had seen both the wizard and the direct Wikitext for the infobox, they universally preferred using the wizard overlay to make edits.

“It’s not clear that this is a pop down for easy editing vs. a drop down for wiki(text) editing” – Caroline, 25, Project Manager

Blue text in capsules indicated to some users that they were external links. Half of the participants in the study were shown template capsules with black text, and half were shown template capsules with blue underlined text. Some of the participants who saw blue text assumed that the blue text indicated a regular hyperlink. They were therefore reluctant to click the capsule for fear of being taken away from the editing task at hand. On the other hand, with black text on the gray background, there is some concern that the capsules blend in to the code too much and do not invite editors to explore them.

“OK. It scared me for a sec that it looks like they were going to hyperlink. Usually when they are blue and underlined... but no, they just go down.” – Bianca, 22, Intern

“You can tell its going to link you to somewhere else because we saw that the links are like that, and the references are like that, so we know that you’re going somewhere else...” – Sara, 21, Student

The wizard icon on template capsules had no meaning to users. None of the users who mentioned the icon had any idea what it might signify. Recommendations: Consider dropping the icon in favor of the word “edit”. The word “edit” will invite editors to click, and will do a better job of conveying the outcome of a click. “Edit” should be displayed in a different style than the title of the template (a different color, font and/or weight) to help distinguish it from the title and increase “clickability.”

Text in templates needs to be searchable. When participants were asked to edit the population of a city displayed in the infobox, one of the primary strategies they employed to find the correct place to edit was to use Firefox’s Find function to search for the word “population”. But unless template capsules were expanded, then the word population did not appear on the page , and so was not found in the search. This completely flummoxed participants. They knew that the word “population” must be present in the Wikitext because they could see it in the Read version of the page, and yet Find couldn’t find it.

Future Work

Overall, we found that the Wikipedia Usability Initiative was successful in the endeavors we tackled to improve the editing process. Namely, the cleaner skin and tabbed navigation made for simpler and faster navigation; the simpler and more graphic toolbar offers more intuitive controls to enter wikitext than the old toolbar; the dialogs proved valuable in generating wikitext; the navigable table of contents both helps people navigate large articles and provides a snippet of real time updates; the integrated help proviced quick and easy access to common tasks (and avoided verbose help pages and navigation to another page); and templates improve the readability of wikitext. From where we started, we have improved the ease of the editing process, however it is still somewhat beyond the reach of an "average user" and there is still much to be done.

The first of our work will include iterating on the features that didn't fully solve the problems we were trying to address.

Templates: As mentioned above, template "capsules" proved very effective and hiding the complex wiki syntax, providing easier access to an article's body content. However, the interaction to access this template syntax needs to be improved. The "pop up" mode was universally preferred for new users, and though we did not test with advanced users, we know through feedback that they still require direct access to this code. We will be working on a solution that hides this wiki syntax while still providing access to the breadth of users that needs to edit it.

Preview and Publish: Users interactions illustrate the need for side-by-side or in page preview. The preview tab that we added worked well, but users struggled to differentiate between the editing and previewing states. We additionally need to refine the flow to not cause the confusion mentioned above to bring users to save their content.

Navigable Table of Contents: The navigable table of contents, as a feature, proved its value in our study and is ready for deployment. However, more QA and browser/os compatibility work needs to be completed before this can happen.

As we move forward, we will also continue to bring the Wikipedia/Mediawiki editor closer to the publishing tools (read: rich text editing tools) that users expect. Additionally, we will also begin to look at the broader user experience and many of the issues we surfaced last March that we could not address in the editing experience alone. In carrying out this work, we have developed and improved the organizational processes, testing, and tools as well as the volunteer community involvement necessary to forge ahead.

Video Highlights

Notes

The team's rough notes from the interviews are here.
gotomedia's Notes