Wikipedia:WikiProject CRUK/Final report 2014-15

Animation 10 Things You Didn't Know About Pancreatic Cancer. Like all the media on this page, released on a open licence by CRUK as part of this project

Final report on period as Wikipedian in Residence at Cancer Research UK, May 2014 to February 2015

Adapted from the end-of-grant report submitted to the main funder, the Wellcome Trust, using their set format

The grant was given to fund a part-time Wikipedian in Residence based at CRUK’s London headquarters. Aims included: to foster collaboration across the charity, the Wikimedia community and the UK cancer research community, with the aim of improving Wikipedia’s cancer-related content for the benefit of all, establishing a robust case study for other organisations, and producing research on how the general public uses cancer information on the internet. John Byrne worked in the Press and Science Information team in CRUK, four days a week, managed by Henry Scowcroft. The Wikipedia account User:Wiki CRUK John was set up for the project, with Wikipedia:WikiProject CRUK as the project page, where much more information can be accessed. The project was extended with additional funding to last 9.5 months, from May 2014 to February 2015.

Outcomes

Objectives as defined at the start of the project

Content: in particular to target the main Wikipedia articles on the four cancers identified as having special unmet needs in the CRUK research strategy: pancreas, lung, oesophagus and brain. To get them to a state where they can be nominated for Featured articles on Wikipedia.
Training: deliver training on editing Wikipedia to CRUK & CRUK-supported researchers.
To get CRUK-owned media onto open licenses for CRUK media, where possible. To upload them to Wikimedia Commons.
Research: two pieces, one qualitative and the other quantitative.

Comments on achievement of these

Content: Largely achieved: Pancreatic cancer and Endometrial cancer greatly improved and newly made Featured Articles (FA). Lung cancer heavily revised and remains an FA. Oesophageal cancer greatly improved, by 424 edits over the WiR period, vs 40 in all 2013, but not yet FA. Less progress made on Brain tumour, a knotty topic with so many types and so few general characteristics.

Many other cancer-related topics improved, including general ones on risk factors. In particular the CRUK Statistics team have added key UK statistics to all 30+ cancers they cover, and have made continuing to do so part of their work routine.

Training: Achieved: The number of sessions and trainees was slightly exceeded, but continuing editing by those trained has been less than hoped, though in line with similar projects.

Media released: Exceeded. 532 media files have been uploaded to Wikimedia Commons on open licenses. Most of these are diagrams, but there are also photographs, informatics and animation. The diagrams have been very widely used by others, mostly on cancer articles. By July 2015, articles using the images had received over 16 million views on desktop, with possibly as much again on mobile & tablet. The images are used in over 20 language versions of Wikipedia.

Research: Achieved. The qualitative research was expanded, and has produced results that exceed expectations, although analysis and writing-up remains ongoing. The quantitative research went according to the plan by the start date, and has produced good results, in terms of showing different ratings for the sites tested.

The nature of the research shifted away from the focus groups in the original grant application to a combination of a qualitative online survey of 1,000 and 30 qualitative interviews. These have almost certainly produced more interesting material than the original plan would have done.

Evaluation methodology used

Content: There were large numbers of comments by other Wikipedia medical editors at the WikiProject Medicine talk page, the talk pages for individual articles, and the review process for FAs (for example the Pancreatic cancer Featured article candidates page had 331 comments by 22 editors, totalling 131,334 bytes.

In addition to reviews by internal CRUK specialists, external reviews were received from leading clinicians and researchers for the target articles approaching FA status. The qualitative research involved getting 2 groups of c. 333 online respondees (via YouGov) to rate “before” and “after” versions of one WP article.

Training: evaluation forms were handed out for most long training session workshops, though not all were completed.

Media released: Mainly by usage statistics, and comments made by users.

Research: The research was planned and carried out in consultation with Dr Henry Potts, Senior Lecturer in health Informatics at UCL (Farr Institute).

Key findings from the evaluation in terms of project development and delivery

Content: Not as many articles as hoped were taken to FA; it was always a challenging target. The ones that were developed can be regarded as a success, though the input from CRUK itself varied by department.

Considerable effort was put into involving the existing medical editing community, with some success, but many editors became involved at the review stages rather than in compiling the content. This resulted in a very stringent review process, and further improvement to the article, but forced the WiR to spend more time than anticipated doing the spadework improving the articles. This was undoubtedly the main reason for the target articles not all being taken as far as hoped. There were competing demands on the medical editing community, not least the Ebola virus epidemic in West Africa, where Wikipedia was a key source of information internationally.

Training: Feedback was generally good but few trained have continued to edit. This is a very common problem in Wikipedia training for new editors. There are other benefits in terms of increasing understanding and raising awareness, and some may later edit, often using different accounts, or without logging on. Specific training to individuals who will, it is planned, continue to upload newly-released media was effective, and by the end of the project some media were being uploaded by CRUK staff, with the WiR just being informed about it afterwards.

Media released: Received enthusiastically by other medical editors. Some 220 images are now used in English Wikipedia articles, and the great majority were added to them by other editors, rather than the WiR and those at CRUK. The page view figures achieved are larger than anticipated, at 900K to 1.4 million views per month of articles containing images (desktop figures only - mobile may be at much as the same again). In total these articles have had over 16 million desktop views in 13 months, mostly on the English Wikipedia, but also in over 20 other language versions.

As a result of the project CRUK has made a very wide commitment to releasing its material on open licenses where appropriate. The standard CRUK model release forms have been changed to allow models/subjects to opt into this in future. Few images containing individuals could be released during the project’s life, as this had not been in place earlier.

Research: The research is still in the process of being written up, but the results seem perhaps more rich and interesting than anticipated. The qualitative piece of research changed and grew in concept during the period, ending as 30 qualitative interviews with members of the public, watching and recording them research pancreatic cancer on the web, and then interviewing them about the choices they made and how they rated the sites they saw (which always included Wikpedia).

Key findings from the evaluation in terms of impact on, or response from, audiences

Content: The quantitative research assessed this, and preliminary analysis has been encouraging. Improvement in Wikipedia articles does not typically produce a great improvement in views in the short term. Many favourable comments were received, many from other editors, some acting as readers for the articles in question, others as reviewers or editors.
Training and events: Ratings and comments generally improved over the period, as CRUK-specific material was developed and practised. There has been incorporation of helping and releasing media to WP into the normal working routine of some several CRUK departments but not others.

In CRUK awareness of Wikipedia among CRUK staff as a heavily-used source of medical information has become much more widespread within the organization, and the large number of page views for articles worked on or containing CRUK-released media have been noted by senior management among others. Apart from the improvement in coverage of cancer in the Wikimedia projects, the medical editing community is more aware of CRUK resources available to them, both online and via email etc. Over 60 CRUK individuals, researchers and specialist staff, were trained to edit Wikipedia. The number that have continued to do so since training is however low, in line with experiences elsewhere.

Whether or not a regular and efficient route of communication between Wikipedia medical editors and CRUK expertise has been successfully established, only time will tell.Cancer Research UK is now actively exploring how best to use the lessons learned during the project in future communications, information and training activities.

Comments on the project from inside CRUK and Wikipedia have been very favourable. Primary audiences are covered in the next section. As regards secondary audiences, one aspect of the quantitative research study was to obtain quality ratings for “before” and “after” versions of the Wikipedia article on Pancreatic cancer, which had been considerably improved as part of the project. The responses, from two groups of c. 340 individuals who each saw just one version, showed general readers appreciated the changes, with, for example an increase to 64.4% from 50.3% agreeing that “The language on the website made it easy to understand”, an increase of 6.5% in reported trust in the information on the page, and so on. The full findings are intended to be published in an academic journal tbc, subject to successful peer-review. If this is not possible, we shall seek alternative means of dissemination, probably via Wikipedia itself.

Media released See above. Many favourable comments have been received; for the viewing figures see the previous section.

Research Preliminary results have been briefly presented at a Wikipedia conference in The Hague, and will be presented again at the Wikipedia Science Conference in September 2015. It iis hoped to publish one or perhaps two papers.

Publicity and dissemination

The project was publicised internally on Wikipedia, and on blogs by CRUK and the Wikimedia Foundation (link?).

There was a CRUK podcast, and newsletters for those involved. There was an interview on MEDSCAPE.com (http://www.medscape.com/viewarticle/830696), and mentions in news stories on the The Independent and the BBC website. There were presentations to two international Wikipedia conferences, and at the Science, Policy, Outreach and Tools (SPOTON) conference in London. More conference talks are planned; one proposal – at the Wikipedia Science Conference - has been accepted. Also a final report for Wikimedia audiences (based on this but more reflective), and blog posts publicizing that.

The main project page, with many links to other pages, remains on Wikipedia at https://en.wikipedia.org/wiki/Wikipedia:WikiProject_CRUK

It is hoped to publish 1 or 2 papers in the future, with the results of the search, with the help of UCL Health Informatics people (Senior Lecturer and MSc students)

Media mentions

Blogs etc.

New images released are quickly put to use - my blogpost re the CRUK diagrams and also some Royal Society images is on the global Wikimedia Foundation blog, and also the Wikimedia UK blog here, 11 September
CRUK science blog post Wikipedia – is it fit for patient consumption?, 27 May
John's blogpost on Wikimedia UK's blog, 20 May, repeated on the Wikimedia Foundation blog, 31 May, Highlights & Global editions
CRUK podcast section, interview with me: section from 6:30 to 10:40

Other media

Presentations at Wikimania 2014

video of: "Wikipedia and Medicine", Speaker: James Heilman (09:10 to 45:00 on file)
"What does a Wikipedian in Residence in the scientific sector do?", John Byrne & Henry Scowcroft of Cancer Research UK, Sydney Poore, Wikipedian in Residence at Cochrane Collaboration. Video on You Tube, starts at 1:10:30
"Medical information online; Wikipedia's place in the ecosystem", panel: James Heilman (Jmh649]/Doc James) MD, CCFP-EM, Clinical Faculty member of the Department of Emergency Medicine, Faculty of Medicine, University of British Columbia (Clinical Instructor), Henry Potts, Senior Lecturer, Centre for Health Informatics and Multiprofessional Education (CHIME), UCL Institute of Epidemiology & Health Care (Bondesgesou), Henry Scowcroft (HenryScow) Cancer Research UK's news and multimedia manager, and John Byrne (Johnbod), now the Cancer Research UK Wikipedian-in-Residence (as Wiki CRUK John). Video here, starts 1:03:20)

Progress reports

Newsletters

Mainly written for CRUK non- or new Wikipedians.

Newsletter 1 September 1 2014
Newsletter 2 October 9 2014
Newsletter 3 February 2015

Impact

Awareness

The project page at Wikipedia:WikiProject CRUK, and its various subsidiary pages, were regularly updated with news, and a number of interim reports were issued, with some newsletters sent to those who had been involved.

Estimates of primary audiences reached

By age

Older Youth (16-19) 5
Young Adults (20-34) 214
Adults (35-64) 47
Adults (65+) 11

TOTAL 277

By gender

Male 129
Female 148
TOTAL 277

By category

Estimated numbers:

Non-specialist adult 63
Academics/Researchers 52
Health Professionals 40
Non-medical CRUK employees) 114
Teachers/Educators 8
Online Many more

TOTAL 277 + online

Further Details: The primary audiences were employees of CRUK or researchers funded by them, and no data was collected for them. The CRUK audiences were over 50% female, and mostly aged 20-34. The researchers varied a bit more. “Under-represented audiences” were not targeted as such. Secondary audiences These “secondary” audiences were really the “primary” ones for this project. The easiest figures to collect are the hits to articles that contain CRUK images released under this project: these are running at about 1.35 million per month (desktop only, uplift by c. 30% for tablet and mobile views), and are likely to slowly increase both by adding to new articles, and organic growth in views. Views to April 2014 total 13.25 million (before uplift). The vast majority so far are to the English Wikipedia, but altogether 22 different language versions of Wikipedia use some of these images. The image files are in svg format, making it easy to translate the captions in new file versions. Much the most-used images are diagrams showing anatomic details, stages of cancer, and features of cells; these are also the majority of the uploads.

The English Wikipedia is accessed by English-speakers globally, including large numbers from countries where English is not an official language (something known but confirmed by our research), especially for medical topics, where the English Wikipedia is more trusted than the local language version by, for example, Poles and Ukrainians.

The Wikipedian in residence (as User:Wiki_CRUK_John) made a total of 2868 edits, but more edits were made by other editors involved in the project. These are hard to quantify, but for example Pancreatic cancer was edited 1367 times (bottom of the page here) in the WiR’s period, against 116 edits in all of 2013.

The page views received by articles whose text was edited and improved by the project are likely to exceed 1 million per month, but this is hard to quantify. The most common cancer types each get 200-600K views per month, on the English Wikipedia, from where content tends to flow to other languages.

Resources developed

Usual Wikipedia training materials used. Special slides developed, which have been shared with CRUK & other Wikipedia trainers.

Ongoing effects

The small funding needed for completing the planned papers arising from the research will be supplied by UCL and CRUK, and the WiR working on a volunteer basis. Aspects of the project, in particular releasing and uploading newly created media to Wikimedia Commons, and updating Wikipedia articles, have now been embedded in normal working routines for some CRUK departments. The medical editors on Wikipedia are now more aware that cancer-related queries can be sent to CRUK, and a dedicated CRUK email address for Wikipedia queries has been set up.

Conclusion

Many thanks to Wellcome for the original grant, and the supplementary amounts from them, CRUK and WMUK. The project was complex, navigating largely uncharted waters, and although the planned timing did not work out, we feel it has had considerable achievements, and been well worthwhile. There has been lasting improvement to Wikimedia content, and processes for continuing engagement set up. Some aspects of the project were pioneering within the context of Wikimedia engaging with institutions, and the results will help other projects in the future.

WMUK comments

Cancer Research Approached Wikimedia UK in April 2013 outlining a proposal for a joint project based on a Wikimedian in Residence project, which in itself followed from a smaller collaboration a couple of years earlier. We have worked together to find external funding, which was awarded towards the end of 2013 from Wellcome Trust. Going through the application process for funding enabled us to become more specific about the objectives of the project, and to capture how our understanding of the goals may have differed. Additionally, because of three partners being involved (rather than just two), a more rigid structure was useful again to keep disciplined about what all organisations wanted.

Documentation was completed in early 2014, and the project started successfully in mid 2014. Although slow to develop initially, the project really took off from WMUK's perspective once cancer information diagrams were uploaded onto Wikimedia Commons and subsequently distributed onto Wikipedia articles. The audience reach was impressive and the upload stands as a powerful case of sharing knowledge.

From a learning perspective, the project showed a need for WMUK to develop more structured training materials that could be used by others. It was also noted that providing more information and structure for the partner organisation at the set up stage of the project is vital, together with catch up and evaluation meetings during delivery stage.

Overall we are very glad to have partnered with CRUK on the Wikimedian in Residence project.