Tracking here refers to techniques to ascertain what use and reuse has been made of open educational resources by people after they have been released. The emphasis is on tracking what has happened to a resource: how many times it has been viewed or downloaded, whether it has been copied to another server, whether derivatives have been made. User activity tracking is not in scope here, other than to the extent that it can be assumed that a user will be doing the downloading, copying and remixing (this is touched upon in the chapter on Paradata).

Why tracking is important

There are several reasons for wanting to track resources, depending on the stakeholder in question.

• Funders (including institutions funding internal OER initiatives) need evidence to show that their money has been spent on resources that are useful and meet a demonstrable need; creators and publishers of OERs who have an interest in continued funding should be ready to respond to this requirement.
• For those with policy and operational remits, tracking can be useful to help assess the potential value of time consuming activities, for example ascertaining whether the time and expense involved in clearing rights to allow reuse and remixing of content is justified.
• Authors are likely to have an interest in tracking as it can show where and how often their resources are reused.  This not only provides evidence of the value of the resources that they create, but also provides pointers as to how content may be expanded or revised.

Of course the assumption behind such reasoning is that resource tracking will actually be able to show evidence of use. However tracking can be something of a double-edged sword, as it may equally show that resources are not being used and that the time and effort spent clearing rights so that they could be edited freely was of little value.  It is also clear that tracking should be subordinate to the release of resources, that is, it is necessary to first solve the problems inherent in releasing resources before they can be. In addition, tracking techniques should not limit the ability of users to access and reuse open resources, nor should it interfere with the benefits that resource publishers may wishes to achieve by releasing content as open educational resources.

Programme approaches to tracking

The initial call for Phase One of the UK OER Programme included the following recommendation¹:

“As far as is possible projects will need to track the volume and use of the resources they make available, because: JISC aims to track the outputs of this programme, contributors benefit from access to usage information, and monitoring usage improves resources and services. Projects should put their own tracking and reporting in place, and be prepared to work with JISC to develop these approaches. To support institutions in this, a range of potential measures of success for OER repositories will be published by JISC early in 2009. This data will be essential in determining whether a longer term programme would be useful.”

To incentivise the collection of usage data, the Programme offered small grants to projects to measure and report on the use of their content. Few projects tackled tracking in any depth, the one or two notable exceptions were the more mature OER projects based at the University of Oxford, University of Nottingham and the Open University. The reasons for this apparent lack of interest in tracking, and in some cases active resistance, are explored later in this chapter.

Techniques for resource tracking

Questionnaires and user feedback

Before considering technical approaches to tracking use and reuse it is worth remembering that there are many approaches to tracking that involve little use of technology, for example,  asking users to fill in feedback forms, gathering feedback through informal communication and formal interviews. Clearly there are problems associated with these approaches; the effort required may not be justified by the number of responses received, or the sample may be biased by “friends and colleagues” and other eager users who are more likely to respond than more casual users.  However, these techniques do make up for what they lack in coverage by increased depth in the data that can be gathered.

Use tracking through web statistics

If one has access to the log files kept by the web server hosting the OERs then monitoring how often those resources are downloaded from that server is routine since analysis packages have long existed for such tasks on any website.

This information may also be available from hosts such as Flickr, YouTube, SlideShare and other web sites for sharing resources, displayed as a count of the number of times a resource has been viewed on that site, downloaded, or embedded into third-party sites. This information is often also available through the host’s API to facilitate automatic monitoring.

Information from the logs includes data about the user’s internet domain, client software and the platform being used to access resources. This may be useful if the number of visitors from a specific user group which can be associated with a domain is of interest, for example if the target audience is not in the UK.

The referrer logs also provide information about the link which sent the user to the OERs being downloaded, which is useful in providing information on who is linking to the resources, and on the keyword terms being used to find the resources in search engines.

However the extensive use of caches, web crawling robots (e.g. used by search engines) and the general impossibility of knowing whether a  downloaded file was actually viewed, render the results suspect for any use other than qualitative comparison between resources from servers where the logging and analysis routines are known to be similar.

Web search

The idea behind using web search as a tracking mechanism is that it is possible to discover where on the open web a resource is available by putting a unique phrase, tag or identifier into the text of the resource and then searching Google or other index for that phrase. The key phrase used should be should be unique to the resource, project or programme depending on the level of aggregation required for the tracking.  The Engineering OER pilot project² proposed a “date-of-birth” code³ for this use made up of three parts: three-letters to identify the project or institution, the date of assignment of the code in format ddmmyy, a unique 5 character string to disambiguate resources assigned codes on the same day. The recommendation was that this should be included in the copyright and licence notice as this text is least likely to be altered. As an alternative to the date of birth code as text, one could use an http URI as an identifier with the advantage that it could act as a link back to the original resource or other useful information, and that people following that link would generate useful tracking information (see refback tracking).

Refback tracking

Refback has been suggested by Creative Commons as a means of reuse tracking⁴. It involves putting a link in the resource being tracked to the site doing the tracking (the two variants are that this may be either the publisher or Creative Commons, i.e. independent and distributed or hosted and centralised). If a curious user follows that link (and the assumption is that occasionally someone will) the tracking site will log the request for the page to which the link goes, included in the log information is the “referrer” i.e. the URL of the page on which the user clicked the link. An application on the tracking site will work through this referrer log and fetch the pages for any URL it does not recognise to ascertain whether they are copies of a resource that it is tracking. It’s worth noting that links clicked in on a secure website (https) do not include referrer information if the destination is not also a secure  https destination⁵. To allow tracking of resources located in secure environments the solution is to also make sure the destination is on https. 

Google Analytics and other online analytics

Google analytics (GA) is the most well known and widely used member of a class of online software services that use a number of approaches to provide an analysis of the usage of web-based resources; a well known open alternative is Piwik⁶. Most of these services, GA included, use a combination of JavaScript code and cookies to log information such as how often a resource is used, some of the characteristics of the user’s web client, and the link the user followed to arrive at that resource, the last of which can also give information about keywords used to find the resource on a search engine. Technically this involves the insertion of some JavaScript tracking code into the pages being tracked, this tracking code includes an identifier for a report that aggregates the statistics from individual resources: these reports can be shared over sites and statistics for an individual resource can be aggregated into multiple reports.

The tracking code only works for HTML documents that the person doing the tracking can edit. Non-HTML resources, those hosted on web content management systems that limit what JavaScript can be embedded into resources, and resources that are hosted on a third-party web2.0 sites cannot be tracked in this way.  There are also legal and ethical implications involved, see below.

URL redirects

The idea of using URL redirection⁷ for tracking is that instead of providing a link that goes directly to a web resource, one provides a link to a service which first logs data for tracking purposes and then redirects the browser to the required resource. Such an approach is common in catalogues and search engines, Google for example uses this technique to track which links from search engine results pages are followed. With the recent rise in popularity of URL shortening services (e.g bit.ly or the twitter t.co shortener), which are in essence just URL redirection services, and growing competition between these services, some have begun to provide tracking functionality which allows their users to monitor the traffic being routed through the shortened URL.

This approach may be applied to OER tracking in two distinct ways: links to the OER may be redirected or links from the OER may be redirected.

It is possible to gather information about how often a resource is accessed by providing a redirected URL as a link to the resource and examining the data gathered through it. This is useful for tracking the use of the resource as a result of specific dissemination or advertising activities (e.g. a listing in some directory or an email announcing the resource). It is less useful for providing general information about how often a resource is accessed,  as the direct URL for accessing the resource will soon become known to people who will then be more likely use the direct rather than the indirect URL.

Information about who is using a resource may be gathered by using indirect links in the resource so that anyone following a link from the resource leaves some tracking information. If a suitable link is chosen this can be a reasonable indicator that the resource is actually being used (as opposed to those methods which only show that the resource has been downloaded). The information recorded may include the web location of the OER, which can be valuable if one wants information on whether OERs are being copied and used from sites other than the original host.

Web bugs and beacons

A web bug, or beacon, is an invisible object that is embedded in content and used to record that a user has viewed the resource.  Commonly these objects are invisible images (e.g. a single pixel transparent gif), that act as a badge, or JavaScript code, which are hosted on the server doing the tracking and which are downloaded when the file is opened. The address for the object is given using an absolute URL so that even when the web page is copied from one server to another the web bug object will be obtained from the original server specified. Thus the web bug object is accessed every time the page in which is it embedded is opened.  Tracking methods such as web usage stats and online analytics, including information gathered by JavaScript and stored in cookies, applied to the web bug provide tracking information for the original page even if it has been copied to a server which the person doing the tracking has access to. The precise URL used to call the web bug may be varied (while still linking to the same object), by use of fragment identifiers of the form #resourceID, so that resource-level tracking may be achieved with a single object. Web bugs can be embedded into resources that can make HTTP requests, typically (but not only) web pages. It doesn’t matter if the resource is viewable on the open web or not, so long as the software used to view it can access the web.

Tracking by web bugs is generally considered to be controversial, partly because it involves silently passing data to a third party (i.e. the person doing the tracking is not the end user or necessarily connected to the site where the resource is hosted), and partly because some high profile uses of web bugs are connected to activities such as checking the validity of email addresses in lists used by spammers (see the section on legal and ethical issues below). There are many uses of web bugs which seem to be widely accepted, for example: the use of JavaScript by Google Analytics amounts to a web bug; the smiley visible somewhere on many wordpress.com blog pages is used for tracking; and, many ‘badge’ or ‘button’ images advertising the use of or support for a service, campaign or software (e.g. Creative Commons) which are inserted by copying and pasting a link to an image that resides on another website, could be used as web bugs for tracking.

A side benefit of this approach is that having a link in the resource to an object on a server under the control of the OER provider may be useful for conveying information after the resource is released. For example, a single pixel transparent gif could be changed to a large red warning indicating that information contained in the resource is no longer valid.

Issues

Appetite for tracking

During the course of Phase One of the UK OER Programme it became clear that tracking was not a priority for many of the projects. Even the offer of additional funding for reports on tracking, during Phase Two of the Programme, elicited few examples that went beyond simple reporting of access statistics such as the number of views and downloads from the hosting web server. Of the two meta studies commissioned by the Programme, the Value of Reuse Study⁸ or the study on Learners Use of Learning Resources⁹, neither identified significant quantitative  data from the projects, though qualitative data was more easily found.

The reason for this may be partially due to a clash in priorities. While funders and other observers had the space to look ahead at how to gather evidence that would be desirable in securing future funding for the sustainable release of OERs, project staff had more immediate concerns. Project teams already had to address many complex issues, e.g. relating to licensing and partnership agreements, before they could start releasing their open educational resources and, clearly, tracking can only take place if content is released in the first place. Also, in the absence of sustained current practice with different approaches to tracking OERS, the long term costs and benefits are unkown. It is arguable that recognition of the importance of tracking is an indicator of  mature OER initiatives, that appreciate of the type of evidence that needs to be gathered in order to make a case for sustainable input and investment.

Technical challenges

Most of the approaches outlined above involve an additional level of technical complexity which could, in certain circumstances, inhibit both the release and use of OERs. For example, third-party redirection services may fail within the lifetime of the resource, thus breaking the link to the content. Unless one has confidence in the longevity of such services, third-party redirection services should only be used for short-term tracking, e.g. tracking the effectiveness of a time limited dissemination campaign. However the redirect need not be handled by a third party; it could be the resolution service hosted by the OER provider for HTTP-URI identifiers for the resource, that are independent of where the resource itself is hosted. In the case of web bugs, as loading them involves an HTTP request and response, which will take time even if the image being used is minimally small, this may be an issue if the server on which they reside is slow to respond.

One of main limitations on how well several of these tracking methods work is the assumption that the resource being tracked will be a web resource: written in HTML, hosted on a web server, viewed and used through a web browser or other internet-connected application. In reality many OERs released through the UK OER Programme were either “old fashioned” formats such as Powerpoint and Word documents, or were multimedia content such as audio or video recordings. While ouch resources are frequently hosted and disseminated from web sites they themselves are not “web native” and their use is more difficult to track.

Legal and ethical issues

There are ethical and legal implications involved with several of the tracking approaches outlined above.  While the emphasis here is on tracking the use of resources, this necessarily involves tracking the action of individual users, which has privacy implications. The EU e-Privacy Directive¹⁰ (“cookie law”) may apply to some of the methods discussed in this chapter. Some users may take steps to avoid being tracked, which will render some efforts at resource tracking ineffective, for example URL redirection is sometimes used for nefarious activities and may be flagged as malware by some applications. Perhaps the biggest cause for concern is that tracking methods, that are used for reasons that some users find objectionable, will lead to reputation loss by association. The distinction between tracking the use of a resource and tracking the users of a resource may not be seen as clear cut by everyone involved.

As Helen Beetham has said “openness is the enemy of knowability”¹¹. Tracking would be much easier in the context of a closed centralised system, that requires users to register before accessing content that can not be re-used or copied.  However open educational resources stand little chance of thriving in such an environment 

It is perhaps understandable that projects that are not committed to tracking, are inclined to suggest that it cannot be done effectively. While it is  true that no single approach will capture all use and reuse of a resource, and that some use will be hidden from all reasonable attempts at tracking, there is still more to be lost than to be gained from making no attempts to track the use and reuse of open educational resources.

 

Future directions

The overwhelming findings of the Programme were that appetite for tracking was low and that approaches to resource tracking in the OER domain were immature. However it is encouraging that the more experienced and mature UK OER projects did appear to appreciate the benefits of tracking their content.  The Open University’s TrackOER project¹², for example, is seeking solutions to support its service development and has made rapid progress in investigating web bugs.

The mechanisms for tracking are developing as a component of the digital business models of the big web players, within constraints set by regulatory frameworks such as Data Protection¹³, the EU cookie directive¹⁴ and other privacy laws. It is likely that the accuracy of resource tracking on the web will improve as companies such as Google, Facebook, etc, invest resources in this area, in order to support their business models.

In parallel, it is possible that increased sharing of open usage statistics will encourage people to be less fearful of such data within the context of an open content. More confidence in capturing and interpreting data may also increase the appetite for tracking. Jorum’s beta dashboard¹⁵, developed in parallel with Phase Three of the UK OER Programme, provides a useful example of open usage statistics,  and hopefully reflects the beginnings of a shift in the perceived value of such data.

Related to this final point are emerging developments in aggregating and sharing usage data. At the very basic level this may involve repository managers sharing data information and files, however there are emerging opportunity for recording and sharing paradata.

 

1. HEFCE/Academy/JISC Open Educational Resources Programme: Call for Projects, http://www.jisc.ac.uk/fundingopportunities/funding_calls/2008/12/grant1408.aspx
2. Engineering Subject Centre, Open Educational Resources Pilot, http://www.heacademy.ac.uk/projects/detail/oer/OER_SUB_Engineering
3. Pearce, R., (2010), Discovering OER re-use and derivative works using DOB codes, http://icesculpture.wordpress.com/make-evolved-oer-discoverable/ ^{^}
4. Creative Commons, Reuse tracking, http://wiki.creativecommons.org/Reuse_tracking
5. Wikipedia, HTTP Referer Hiding http://en.wikipedia.org/wiki/HTTP_referrer#Referer_hiding
6. Piwik, open source web analytics, http://piwik.org/
7. Wikipedia, URL Redirections, http://en.wikipedia.org/wiki/URL_redirection
8. White, D., and Manton, M., (2011), Open Educational Resources: The value of reuse in higher education, http://www.jisc.ac.uk/media/documents/programmes/elearning/oer/OERTheValueOfReuseInHigherEducation.pdf
9. Bacsich, P., Phillips, B., and Frank Bristow, S., (2011),  Learner Use of Online Educational Resources for Learning,  http://www.heacademy.ac.uk/assets/documents/FLearner_Use_of_Online_Educational_Resources.docx
10. EU Directive 2002/58 on Privacy and Electronic Communications http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:32002L0058:en:NOT
11. Beetham, H., (2011), Understanding the role of OERs in open educational practices, http://www.slideshare.net/loumcgill/understanding-the-role-of-oers-in-open-educational-practices
12. TrackOER, Open University, http://cloudworks.ac.uk/tag/view/TrackOER
13. UK Government Data Protection Act 1998, http://www.legislation.gov.uk/ukpga/1998/29/contents
14. EU e-Privacy Directive, http://www.ico.gov.uk/for_organisations/privacy_and_electronic_communications/the_guide/cookies.aspx
15. Eagen, L., (2012), Collecting statistics just got a whole lot sweeter, http://www.jorum.ac.uk/blog/post/38/collecting-statistics-just-got-a-whole-lot-sweeter