Usability and Experience Study

The Wikimedia Usability Initiative Team partnered with Bolt Peters, a San Francisco, California, user experience consulting firm, to conduct both in-person lab and remote interviews and studies of the general Wikipedia experience, with focus on the editing experience, process, and its successes and failures. The study consisted of 15 one-on-one interviews, each lasting between 45 and 60 minutes. 10 of these interviews were conducted in person at a lab facility and 5 were conducted remotely from the Wikipedia Usability Office. The interviews were conducted in late March 2009 in San Francisco, CA.

The primary research goals were to:

identify obstacles that novice users encounter in editing a Wikipedia article—including, but not limited to—adding personal content, fixing a typo, adding a reference, and contributing to discussion pages
identify obstacles in creating a new article
evaluate the self-sufficiency and legibility of help materials and documents found on Wikipedia.org
evaluate how novice users interact with templates
discover user experience patterns and issues that have not been previously identified.

Methodology

Target Audience

The target audience of the Usability Study was limited to regular Wikipedia readers who are willing to contribute their knowledge to the site, but have expressed reservations about doing so. The team aimed to select a majority (~80%) of users that had not edited but were willing to, and a minority (10%) of users that had not edited and weren't willing to and (10%) of users that were novice but not new editors, with less than 5 contributions. We preferred potential subjects whose primary reasons for not contributing seemed to be due to the technical complexity of the interface of MediaWiki and markup, but understand that fear of article deletion, unwillingness to have work edited by others, lack of confidence in 'expertise', not being willing or used to collaborating, and philosophical differences are all also big inhibitors to Wikipedia readers' contribution.

According to a multilingual survey of Wikipedia readers conducted by the Wikimedia Foundation and UNU-Merit, 74.6% of Wikipedians are male, and 24.7% are female. The average age of Wikipedians (aged 10–85) is 25.1 years old (the average male Wikipedian is 25.6 and the average female Wikipedian is 23.7. Since the team's goals are aimed at reducing common barriers for Wikipedia readers to become editors, and there are over 300 million unique Wikimedia visitors, the majority of which are English Wikipedia readers, we set out to keep our gender (M/F) breakdown equal and our gender spread across ages (Under 18, 19–34, 35–44, 45–54, 55–64, 65+), so that our research would give us insight into the entire spectrum of potential new editors.

Our desired participant profiles:

Participant No.	Has not edited, but is willing to	Has not edited, doesn't see themselves editing	Has made < 25 contributions	Male	Female	Under 18	19–34	35–54	55+	Uses WP everyday	Uses WP > Once a week	Uses WP Once every few weeks
01	X				X	X				X
02	X			X			X				X
03	X				X		X			X
04	X			X			X				X
05	X				X			X		X
06	X			X				X			X
07	X				X				X	X
08		X		X					X		X
09			X		X		X	X			X	X
10			X	X				X	X		X	X
11	X				X	X				X
12	X			X			X				X
13	X				X			X		X
14	X			X					X		X
15			X		X	X	X	X	X	X	X

Lab Testing

The Lab testing was conducted at the Fleischman Field Research facilities, located in San Francisco, California. Participants, limited to those within driving distance, were brought into a room with the Research Interviewer, and a Mac or PC, based on their stated preference. Behind a two way mirror, Wikimedia Usability Researchers could view the interviewer and participant interacting, hear the interview audio, view the participants entire screen and computer interactions, and see their faces and expressions, in real time as the interview was being conducted. These videos (both the screen and the participant) were also captured under a Creative Commons License and can be viewed in the section labeled "Full Interview Videos".

Remote Testing

The Remote Testing was conducted at the Wikimedia Usability + Wikia office. Participants were recruited live, and called for interview within 1hr of the completion of their survey response. They completed the script in their native environment. Using UserVue, a remote screen sharing application, the B|P interviewer, the Wikipedia Usability Team, and other observers could view the participant's screen in real time, and hear the complete audio via telephone and speakers. The audio interview along with the screen capture videos were recorded and are available here under a Creative Commons license (see "FullInterview Videos below.")

Lab Testing and/VS Remote Testing

We decided to do a combination.

Users from outside the San Francisco area
In person—see more

B|Ps Lab Vs. Remote Comparison

Recruiting

We recruited over 2,500 San Francisco residents for the in person testing. A banner/alert was placed and displayed at the top of every 1 in 100 Wikipedia pages for the duration of 8 hours. When users viewed our message by clicking, they were forwarded to B|P ethnio recruiting system where we asked them a series of questions. Specifically, users were asked about what they were doing on Wikipedia, how often they used Wikipedia, if they had ever contributed to Wikipedia (and in what manner), their age, gender, location, and availability.

Based on these criteria, the 2,500 users who responded to our survey were filtered down to 500 viable subjects based on their answers to these questions. The team, along with B|P, partnered with Davis Recruiting to contact, filter, and screen these 500 participants based on their Wikipedia contribution history, Wikipedia usage patterns, their given reasons for not contributing, and their talkativeness and openness to discuss their thoughts and actions. From 2,500 users, we ended up with 10 study participants and 3–5 waitlisted participants.

Screenshot of Ethnio screener invitation

Study Participants

Lab Testing

Participant No.	Age	Gender	Wikipedia Editing History	Wikipedia Usage	Occupation	Other Info	Location
01 "Suzanne"	50 (45–54)	Female	Has not contributed, but is willing to. Has account.	Every day	Paralegal.	Was checking a reference.	San Francisco, CA
02 "Grace"	24 (18–34)	Female	Has not contributed, but is willing to.	Every day	Medical Student.	Was looking up diseases.	San Francisco, CA
03 "Tito"	21 (18–34)	Male	Has not contributed, but is willing to.	Every day	College Student and Video Producer.	Was researching corporate videos.	San Francisco, CA
04 "Dan"	41 (35–44)	Male	Has not contributed, but is willing to. Has account.	Every day	Programmer	Was reading about DVI and HDMI interface info.	San Francisco, CA
05 "Seamus"	43 (35–44)	Male	Has not contributed, but is willing to.	More than once a week	Controller for a non-profit	Was researching for a presentation.	San Francisco, CA
06 "Saurab"	28 (18–34)	Male	Has not contributed, but is willing to. Has account.	More than once a week	Retail Software Developer	Was reading about supply chain processes.	San Francisco, CA
07 "Gene"	64 (55–64)	Male	Has not contributed, and is NOT willing to.	Every day	Retired Public Transit worker	Researching.	San Francisco, CA
08 "Claudia"	64 (55–64)	Female	Had edited fewer than 5 entries. Has account.	Every day	Database administrator	Looking up Frank Loesser	San Francisco, CA
09 "Galen"	24 (18–34)	Female	Has not contributed, but is willing to.	More than once a week	Costumer	Was finding out about Americans of German descent.	Oakland, CA
10 "Nikki"	16 (Under 18)	Female	Has not contributed to Wikipedia, but has to other Wikis.	Every day	High School Student	Was looking up a TV Series.	San Francisco, CA

Remote Testing

Participant No.	Age	Gender	Wikipedia Editing History	Wikipedia Usage	Tech Savvy	Other Info	Location
01 "Lisa"	(55–64)	Female	Has not edited, but is willing to	More than once a week	2: Uses computer to access internet and email apps.	Research	Eureka, CA
02 "Shaun"	(18–24)	Male	Has not edited, but is willing to.	Every day	3: uploads and downloads files from internet and devices	Research	Rexburg, ID
03 "Jerry"	(25–34)	Male	Has not edited, but is willing to.	Every day	2: Uses computer to access internet and email apps.	Reading about the history of New York	New York, NY
04 "Bryan"	(25–34)	Male	Has not edited, but is willing to.	More than once a week	3: uploads and downloads files from internet and devices	Researching psychology concepts	Apex, North Carolina
05 "Carrie"	(18–24)	Female	Has edited fewer than 25 entries	Every day	3: uploads and downloads files from internet and devices	Looking up information	Bayside, NY

Scripting

The moderator script was drafted by the Wikipedia Usability Team, B|P, and the Wikimedia Foundation, and an almost identical script was used for the lab subjects and the remote subjects, the discrepancies being entirely related to the difference in set up. The script aimed to see how users would attempt to and feel about a collection of tasks or objectives that first time and early Wikipedians frequently encounter in their editing process including, but not limited to:

Finding and using the various modes of editing ("edit this page" vs. section edits)
Adding personal content or information to an existing article
Looking at Discussion Pages
Adding comment/thread to a discussion page.
Adding or using content from an external source (website, paper, etc)
Adding a reference to content
Adding an external link to a list or section of links
Editing an article that uses a template
Navigating through template syntax to edit an infobox.
Formatting content (Headers, bold, italics, linking, tables)
Creating a new article
Fixing a typo
Finding and using help

Though the script was generally task based, the questions were open ended and left room for interpretation, encouraging subjects to not try to complete the tasks in any "correct" way, but rather to go about them naturally, if they had felt the motivation to do so on their own. Open-ended interview questions including "How did you find that process? What did you like or dislike about it?" regularly follow every "task". The script was piloted on two internal members of the Wikimedia Foundation—one regular editor, and one newbie editor—prior to the lab and remote testing.

View the complete Moderator Script here
View B|P's explanation of Moderation Techniques here

Summary of results

The primary goals of our study were to examine the ease of use of the Wikipedia editing interface, focusing on both cognitive and UI obstacles to editing and factors that could potentially increase users' editing of articles. We focused on editing tasks, but along the way learned about much more than the specific tasks and the editing experience.

While the team observed no patterns or distinctions based on gender alone seen during the study, we observed patterns amongst the different age groups. Younger users (Under 18–34) tended to have less inhibition to make smaller changes to Wikipedia than older users (45+), and in doing so also tended to learn more by example and employ a trial and error method to making changes. When the trials of these younger users did not reach success, they were also more resilient in recovering from those errors. Older users tended to give up more readily, feared they might "break" things, and often sought instructions from the "they" behind Wikipedia.

People Love Wikipedia (and get there easily)

"I use Wikipedia all the time. Usually it’s the most information in the easiest spot to access. It always looks very well put together it boggles my mind how many people can contribute and it (still) looks like an encyclopedia." -- Galen, 24, Costume Designer

"I like Wikipedia because it's plain text and nothing flashes" -- Claudia, 64, Database Administrator

If there was one thing that was consistent and unanimous across our study participants, it was the assessment that Wikipedia is an incredibly valuable information resource whose accessibility is unparalleled. Aside from its value as a reference, a time and lifesaver, and an up to the minute news resource, participants also praised its simplicity, coherence, and breadth. They find that the quality and quantity of information they find there is outstanding, trustworthy, and their go-to reference when they have a question or need more information about a topic, even one that they themselves might have quite a bit of expertise on. Users also tend to click on Wikipedia articles if they see them as part of the results list when they search for a topic/subject, because they trust the content and are familiar with Wikipedia. An extra thanks to Google, for showcasing just how referenced Wikipedia articles are—consistently making their links one of the top hits—our users count on that!

"I usually use a Wikipedia shortcut on my computer" -- Suzanne, 50, Paralegal

"It's usually [one of] the first hit[s] on Google" -- Grace, 24, Medical Student

"I don't usually go to the Wikipedia front page." -- Dan, 41, Programmer

By and large, participants explained that they accessed Wikipedia articles through a top search hit from Google. Other methods of entry were through browser search plugins, shortcuts, and referring articles. When asked to head to Wikipedia, many participants stated or explained that they had never been to the front page before.

Wiki Syntax is not that hard to ignore (in small doses)

"So now I learned how to edit. It was okay, just like editing, I do it every day…. Things like my personal profile on an internet dating site." -- Gene, 64, Retired Public Transit Worker

"Pretty cool, easier than I thought, I just have to get used to it." -- Lisa

Users' word processing skills translated to the task of editing text in Wikipedia so that all the users we spoke to, including the technically uninitiated, were able to identify and make changes to the content text (as opposed to the wiki markup syntax) on Wikipedia. The changes users were able to make included correcting typos and grammar, adding words and sentences, copying and pasting, and formatting text (with bold and italics). The ease of editing around wiki markup drastically decreased as the complexity of the article increased—most notably when articles started with large infoboxes, templates, and other syntax-heavy elements.

The "right" way Vs. The "wrong" way

"What I did was a hack, I'm not actually using the site" -- Claudia, 64, Database Administrator

"Rather than making a mess, I'd rather take some time to figure out how to do it right" -- Dan, 41, Programmer

All of our participants are Wikipedia readers, but had little or no experience with editing. Generally the editing process was not a warm and welcoming one. Before subjects even hit the ‘edit’ or ‘edit this page’ buttons, they voiced concerns about the rules, proper etiquette, formatting, and were naturally conscientious of and inhibited by maintaining the community expectations. When a few of them attempted to find answers to their questions about rules and etiquette, they were overwhelmed with the amount of information and documentation they encountered.

The most commonly expressed concerns were around who was allowed to edit what, who reviewed the edits, and what level of privileges were given to which users. Users also expressed concern and hesitation about editing without doing extensive research and feeling confident in their ability (put somewhat on the spot) to add relevant and accurate information. Not knowing what they could or should add/edit is a large barrier to at least some users' willingness to edit.

Another major concern that participants expressed was the need to cite and validate their information. More often than not, our subjects did not know when they were expected to add a reference, felt an impulse to add a reference anyway, and faltered on the right way to do that, which included determining where in the article (or which section) was appropriate, the correct formatting, and the appropriate syntax when the references were autogenerated.

In cases such as these references (as well as html links, internal links, and lists) many users copied and pasted text from examples from within the same article in order to make their edits appear the way they wanted them to—namely to appear like others on the page. While this worked fine for most simple, basic edits, users felt that this was a ‘hack’ and they weren’t doing it in the ‘right way’.

The final observed example where participants struggled with the right and wrong way to do things was when they received conflicting messages. One user whose edits were immediately reverted after they saved was flustered and concluded that edits must go through a review process and assumed they'd 'show up in a few days.' Additionally, when viewing discussion pages participants felt quite confident about what type of content and discussion was appropriate, until they encountered the most noticeable text on the page stating "this is not a forum," after which the doubts started to roll in.

Feeling Stupid

"[I felt] kind of stupid." -- 24, Galen, Costumer

Every user in this study struggled to get a basic grasp of the editing interface. Despite users’ overall excitement about Wikipedia, their willingness to spend up to an hour on the site, and varying levels of computer expertise, they largely failed to make edits correctly without repeated attempts and efforts. Users regularly commented that they had ‘no idea’ or ‘no clue’ what they were looking at, or what they were doing. While they actually were able to edit simple text, users were not at all confident that they were succeeding during the process.

As users edited, they mentioned that they thought they were looking at html or some variation of html, and they all admitted that they weren’t familiar with it, or knew very little. They commonly used descriptors such as ‘computer stuff’, ‘technical stuff’, ‘programming language’, ‘syntax’, etc.; in other words, something that is ‘not for me’, but for someone who has been trained in this professionally. Many users described their ideal or expected interface as something more like facebook.com, myspace.com, or a blog- an intuitive GUI that hides the code.

This led to intimidation in editing anything, ‘in case I mess it up’. Users were not confident that they were doing the right thing until they had previewed the page and seen that their changes were displayed in the way they expected. Users also did not find the toolbars too helpful- though nearly all of them hovered over the toolbar icons to read the pop-ups, the only icon they recognized and used consistently was Bold.

Expertise
Level of comfort
Lack of consistency
Conflicting messages
10 ways to do 1 thing

Feeling Overwhelmed

"There sure is a lot of stuff to read" -- Dan, 41, Programmer

"This is where I'd give up" -- Claudia, 64, Database Administrator

"I'm sure there is help out there, but I am too lazy to read that." -- Tito, 21, Student and Video Producer

"What's to stop you from just putting in anything?" -- Suzanne, 50, Paralegal

"But it's on the internet and people will read it and believe it's true." -- Grace, 24, Medical Student

There were a variety of things that left our participants feeling overwhelmed, but articles that started with infoboxes, thick wiki syntax, help docs, and any attempts to find concise answers on rules or guidelines left participants feeling lost. Some explained "If [I] can't read it in five seconds, you've lost me," while others exclaimed "It may take a whole night to do this", and another spent the majority of their time with us searching and digging, explaining "If I really wanted to put it on there, I'd find a way to do it." While their thresholds varied, the results were strikingly similar—lots of information, not a lot of guidance; more questions, fewer answers.

What I see Vs. What I get

"I'm more of a visual person." -- Galen, 24, Costumer

"I couldn’t really understand the format, I didn’t know what it was saying. I would just go to the stuff that’s readable. It looks kinda like a website, lingo stuff." -- Tito, 21, Student and Video Producer

"In many websites, you kind of see the screen just the way you see it in the article, Here it looks like they converted it into plain text. I think what I’ll have to do is open another Wikipedia, so I can compare the views. In blogs, it’s easier to add stuff- you don’t go into the programming mode. This html version- its much easier to edit a blog." -- Saurab, 28, Retail Software Developer

Once within the editing environment, most subjects commented on the illegibility of the hybrid Wiki syntax and article content—the more complex the article, the more exaggerated the response. When users made it past their initial reactions, navigating around the syntax to perform basic word processing tasks (correcting a typo, inserting a block of text, bold and italics formatting) proved less problematic than finding a particular section, adding references, using tables, creating and naming links. But not even our youngest and most computer savvy participants accomplished these tasks with ease.

Aside from feeling confused by the “code”, “computer lingo”, and “html”, subjects could not correlate what they were seeing within the edit box to what they saw on the article page. We saw the vast majority of participants perform the same behavior as they began to edit—most subjects opened a separate browser window to view the static article as they were making their changes and used preview and save before they had finished their work to monitor their editing progress and results.

Expectations
Blogs
Word Processor
Previewing and Saving
Lingo/Html/Lingo

What I do Vs. What I get

"They should have instructions for what you want to do, but it's really not there." -- Suzanne, 50, Paralegal

"Can't something auto correct my links [to internal pages]?" -- Grace, 21, Medical Student

"Let someone else do it (waves hands)" -- Claudia"

A significant number of our participants, noticeably female, tended to have expectations around actions and processes that could happen without being explicitly requested. These users expressed expectations for help of the right variety being presented to them at the right time, automated clean up and formatting of edits ('I hope that someone will come after and fix it'), and the existence of an editing or reviewing staff, mysteriously called "they" (there is no such person or persons!).

Expectations: automation and "they"
Instructions

Exploring (learning by example) and Receiving Instructions (learning from Help Docs)

"I don't know by looking at it [here], but by looking at other examples." -- Grace, 24, Costumer

"I like to learn by example because you don't have to digest a whole lot of irrelevant stuff." -- Dan, 41, Programmer

"I want to see instructions for the "edit this page." -- Suzanne, 50 Paralegal

When participants were asked to perform a task or format a particular sentence, rather than consulting help, they chose to look within that section or article for another example of a similarly formatted word or sentence. This proved to the be the most effective learning tool for uninitiated editors. Copying and pasting from another example relieved some of their concerns about the "right" or "wrong" way to do things and the illiteracy of wiki markup. It also helped in associated what they were seeing in the edit box with what they were typing (or here, copying) within the edit window—"I don't know by looking at it, but by looking at the other ones."

Aside from learning by example, most participants successfully learned by trial and error. Previewing and saving after the most minor of changes to verify the result with the expectation (i.e. correctness) was commonplace. Many participants started off with baby steps—"I'm just going to try and type sort of what I see, just as a test" on an article, and notably not in the sandbox. This method seemed to lower barriers to editing and thresholds to learning—"I can guess on how to do things based on what's already in the page."

Sticklers for Details

"I would correct people's grammar errors, subject verb agreement" -- Gene, 64, Retired Public Transit worker

At the conclusion of our interviews, we asked all of our participants what might encourage them to contribute more to Wikipedia or in what ways they felt they would contribute. Many mentioned that fixing typos and grammatical errors would be one of the actions they'd be most likely to do. In fact without prompting, some users started to do this on their own in the course of completing other tasks. Participants of all of the age groups did this, but those that explicitly mentioned it tended to be older (44+).

References

"Should[n't] I cite the text that I inserted from Creative Commons." -- Nikki, 16, High School Student

"I'm aware that copying directly from another source is discouraged and that references are needed... I don't want to put in incorrect information, that's one of the main reasons I don't go changing stuff." -- Saurab, 28, Retail Software Developer

"How will people know it's genuine information"-- Dan, 41, Programmer, who spent the majority of our session going to great lengths to verify the validity of his edit.

Without any prompting, the majority (one less than the entirety) of our participants expressed the desire or obligation to cite their sources, provide references, or validate the information they were adding or editing on Wikipedia. That desire, however, rarely resulted in a successful citation. Subjects struggled to differentiate between a reference, an internal link, and an external link. In fact, only one subject used the "shortcut" for an internal link in lieu of using the full html address for the forwarded article.

Whether in the course of attempting to create a reference or while accomplishing another editing task, many users hovered over each of the toolbar icons in search of a solution. If/when users found the "reference" button (at the far right!), most could not digest the syntax that was placed at their cursor and questioned where the super script, footnote number, and text belonged and would end up. In the few cases in which the subject was adding a reference to an article where these references were auto generated, the users were completely dumbfounded.

Creating a New Article

The task that nearly every user failed at was creating a new page. Participants passed over the pointer to create a new article within search results, did not associated red links with yet-to-be-created pages, and could find no obvious button or action in the Wikipedia navigation. Several users, while scanning the pages to try to figure out how to create a new article, saw ‘create a book’ on the left, and thought maybe they should ‘add wiki page’ in order to create a new article. The wording ‘create’ likely points them towards that link, as there is no other mention of ‘create’ on the page.

The few users that managed to create a new article commented on the lack of formatting or templating guidelines and expressed surprise that the starting point was a completely blank slate.

Where do I edit?

Users often missed the ‘edit’ buttons next to each section, clicking on ‘edit this page’ all the way at the top. This often got them lost if they were editing a particularly long article, as they weren’t easily able to find the section they wanted to edit. Several users also clicked on the wrong ‘edit’ button next to the sections, thinking that the ‘edit’ button below the section referred to the section above.

Users who got used to editing single sections (instead of the whole page) expected to be able to edit the first section, as well as the template box at the top, separately from the whole page.

Clutter

Repeatedly, users missed messaging displayed on the pages. The amount of information and the number of ‘boxes’ on each page may have distracted users from the most important/relevant messages. Some users also missed the ‘save’ button because they got confused by the double scroll bars on the edit page. The message under the ‘Discussion’ section for various articles was confusing to some users, especially when they read ‘this is not a forum’, which seemed to contradict their thoughts about what a ‘discussion’ page would include.

Help needs some help

"I have no idea how to do that, maybe there is a how to?......Help should be on the top right. It's always there." -- Claudia, 64, Database Administrator

"There should be instructions here. It's not immediately clear." -- Suzanne, 50, Paralegal

While most participants said they probably wouldn’t go there on their own, once they were prompted to explore the Help section, most users did not find the section very helpful, if they even found it. The amount of information is overwhelming to users, and users sometimes got lost trying to find the one set of instructions they needed in order to complete their task. The cheat sheet was the only item in the help section that led to a subjects successful edit. Copying and pasting from other examples within the article or from another article was more often the help users sought.

Additionally, we never saw the same click path through help twice, even when participants were looking for the same information. Users looking for the same information ended up in any number of final destinations (including the "Simple English Wikipedia" when one user was particularly challenged and was looking for a 'simple' answer), the majority of the time without answers. Help proved to be neither intuitive or consistent.

Inhibitors to Editing

Most users said they had ‘thought about’ editing Wikipedia, but never done it because of laziness or lack of time. The few who had edited had made only minor changes to sentences (fixing typos or grammar, etc.) Users mentioned as they tried to edit that they would need ‘more time’ to learn how to do what it was they were attempting, because they didn’t understand Wiki-markup, and it wasn’t immediately obvious what they needed to do. Users also expressed concern that the content they were adding was ‘correct’—in terms of content appropriateness, accuracy, and formatting.

ETC

When using Wikipedia's own search, users rarely saw the "did you mean" results that were close approximations of their search terms, thus having to search again. In two cases, users left Wikipedia's search and used Google instead.
Some users missed the ‘spam-blocker’ code that they had to put in because it appeared above or below the fold of the page they ended up on after they tried to save.
Most users thought the ‘history’ section of the articles would be useful, but only for articles that they had a particular interest in or were editing themselves. However, some users were confused by what ‘prev’, ‘cur’, or ‘hist’ meant.
While most users did not actually try to upload images, the few who did were confused as to how to do it, and how exactly the process worked.

Highlight Videos

Full Interview Videos