Using Yahoo Pipes to redirect to Open Search metadata page instead of intraLibrary public URL (and the power of Twitter)

First of all, a big thank you to Owen Stephens – @ostephens – who responded to my musing tweets on RSS by assembling a Pipe that “Rewrites Intralibrary RSS feed to use ‘link’ to metadata rather than object”; a great example of the power of Twitter for anyone who still thinks it’s an exercise in pointless self-revelation, full of trivial noise.  As Amber Thomas –@ambrouk – put it recently and as I also to tend towards, Owen exemplifies “whole person Tweeting” not restricting our interraction on Twitter to our professional sphere but filling it with more personal and sociable “noise” – the closest thing to a virtual office you will find.  I’ve never met Owen in real life but I shall certainly buy him a pint if our paths ever do cross!

As anyone who has passed by these parts before will know, we have been wrestling with intraLibrary for about two years now to develop a blended repository of Leeds Met’s research output (both Open Access full text and citation only) and Teaching & Learning material (both Open Educational Resources/material for federated access only) and we have spent a lot of time developing the IRISS SRU interface as a front end to provide appropriately differentiated Open Access to the different types of resources.

One of the simplest ways for a repository to alert users to new content is via RSS and it is very easy to generate a feed for pretty much any criteria in intraLibrary; I have generated several feeds for both research collections (by faculty) and for OER.  There are, however, two main issues with these feeds:

• The first problem is associated with the metadata template I have implemented for research and the lack of flexibility to customise which fields are exposed via RSS – I haven’t yet got a solution to this issue.
• the second problem, however, arises because the URL exposed by the feed points to the public URL generated by intraLibrary whereas I need it to point to the Open Search metadata page and this is where Yahoo Pipes can come in.

I don’t have much experience with Pipes but it is billed by Yahoo as “a free online service that lets you remix popular feed types and create data mashups using a visual editor. You can use Pipes to run your own web projects, or publish and share your own web services without ever having to write a line of code.”

Owen’s pipe has three components:

• “Fetch feed” which is simply the intraLibrary generated RSS feed
• “Regex” which applies a regular expression to an item attribute.  In this case it takes the components of the public URL in item.guid.content – oai:com\.intralibrary\.leedsmet:(.*) – and replaces it with the components to build the URL of the Open Search metadata page – http://repository.leedsmet.ac.uk/main/view_record.php?identifier=$1&SearchGroup=Open+Educational+Resources
• “Rename” which does what it says on the tin and simply renames or copies item atributes – in this case item.link becomes objectlink and item.guid.content becomes link

The Pipe Output can be subscribed to as RSS which gives us a feed that does indeed link to the Open Search metadata page rather than the intraLibrary public URL:

http://pipes.yahoo.com/pipes/pipe.run?_id=5c085e83cb144f9a1796558fa7d6d253&_render=rss

Simple when you know how!

N.B.  Some of these links actually DON’T work and I haven’t yet been able to figure out quite why.  As far as I can tell the affected resources were all uploaded as part of the Reproduce project and the links are to none-existent unique identifiers e.g.  http://repository.leedsmet.ac.uk/main/view_record.php?identifier=1432&SearchGroup=Open+Educational+Resources .  I think it may be because some of these records seem to have been ascribed 2 unique identifiers – this is an automatic process in intraLibrary and configured as uneditable so I’m not sure how it has happened.  However, they were originally uploaded by another user before the user profiles and metadata template for ukoer were fully configured and I may need to delete and re-upload as I have done already with http://repository.leedsmet.ac.uk/main/view_record.php?identifier=1673&SearchGroup=Open+Educational+Resources – I’m not sure how/when this will be updated in the feed and currently it’s still linking to the none-existent UID; it may take a little time to update.

Anyway, I’ve copied @ostephens Pipe – hope you don’t mind Owen, I couldn’t find any rights information(!) – and replaced the feed with one of my research feeds – Carnegie Faculty of Sport and Education – and modified the “Regex” module to redirect appropriately by replacing oai:com\.intralibrary\.leedsmet:(.*) with http://repository.leedsmet.ac.uk/main/view_record.php?identifier=$1&&SearchGroup=research.

N.B. As mentioned, there is an additional problem associated with the research metadata template and the lack of flexibility to customise which fields are exposed via RSS – the only way I have been able to accommodate all of the information required for research (using EPrints software as a template) with the intraLibrary metadata schema (UK LOM) is by using multiple description fields as extensions; ideally I would like the abstract to be exposed via RSS but this is in second description field whereas it is the bottommost description field that is exposed via RSS which generally contains whether the resource is refereed or not – this is not terribly relevant so I’ve added a Mapping to the “Rename” module to remove it which means the feed now exposes just the title which does indeed link to the Open Search metadata page:

http://pipes.yahoo.com/pipes/pipe.run?_id=286bbb1d8d30f65b54173b3b752fa4d9&_render=rss

Advertisement

A better beta – Open Search Version 2.0

After mounting pressure from the University, we have now added live links to http://repository.leedsmet.ac.uk/main/index.php from Library Online and the research web-pages but it is important that folk are aware that it is still very much a beta implementation – I have tried to summarise ongoing development requirements below (both for research material and Open Educational Resources) and I am very keen to receive user feedback to inform ongoing development; I also plan on doing some formal user evaluation in the New Year.

The two main issues we have faced throughout the development process is that intraLibrary is designed as a Learning Object repository requiring user authentication. This means we have needed to develop an appropriate metadata template for research material (mapping UK LOM onto Dublin Core) and develop an external, openly accessible search interface to query that metadata (via SRU) and display results appropriately.  Though there has been consultation throughout development I am still conscious that, *potentially*, there are issues outstanding that need to be more widely considered, especially as we move to establish a workflow involving more staff to populate the repository; obviously any changes to the metadata template will carry a more significant overhead the greater the number of records there are.

For research material, the workflow is currently configured in two stages and the approach we are currently considering would require me/designated administrator to complete the metadata in Table 1 and for trained cataloguers to complete the cataloguing information in Table 2:

---

Table 1 (primary metadata entry)

Metadata element	Comments
Faculty	6 Leeds Met faculties (& Thesis/dissertation)
Title	Article title
LOM identifier	Unique identifier applied automatically by intraLibrary
Digital Object Identifier	(if available) Utilising additional (editable) instance of LOM identifier field
Bibliographic metadata: Source title Journal volume Journal issue Source publication date Start page End page	It may not be necessary to complete all of these field depending on the “Type of resource”; they will all be required for a peer reviewed journal article but a book, for example, might only require “Source publication date” to be completed while a book chapter might require “Source title”; “Source publication date”; “Start page”; “End page” N.B.  A related issue that needs to be considered at this stage is that for records entered so far I have been inputting year only to the “Source publication date” field – however, we will need to differentiate records by census period; this perhaps could be achieved with a more specific date in the “Source pub date” field or we could explore using an additional field to record this information.
ISSN/ISBN*	Description field 1/4 – can’t really be differentiated from other description fields for search by SRU
Abstract*	Description field 2/4 – can’t really be differentiated from other description fields for search by SRU
Published/Not published*	Description field 3/4 – can’t really be differentiated from other description fields for search by SRU
Refereed/not refereed*	Description field 4/4 – can’t really be differentiated from other description fields for search by SRU
Author	System bug meant this could not be independently queried by SRU until recently – issue should now be resolved but not yet integrated with interface
Publisher	System bug meant this could not be independently queried by SRU until recently – issue should now be resolved but not yet integrated with interface
Contribution date	Automatically completed
Technical format	Always PDF for research
Type of resource	Scholarly text/ Book/ Book chapter/ Edited book/ Book review/ Miscellaneous conference item/ Conference contribution/ Journal item/ Electronic journal/ Electronic newspaper article/ Report/ Confidential report/ Submitted journal article/ Thesis or dissertation/ Working or discussion paper/ Lecture transcript NB. It is relatively straightforward to add terms/modify existing terms though there may be work involved modifying records. For example, it has been suggested that the vocabulary term “Journal item” should be changed to “Journal article” and while this is relatively straightforward, there is no way of making a global change without reviewing each record individually – currently 218 items.
Statement of Copyright and Restriction	Where a record is citation only (i.e. no full text available) I am using this field to record information from SHERPA/RoMEO to indicate whether I should pursue the full text (for my reference only – i.e. not displayed in Open Search interface).  When there is a full text available this field will generally record that it has been uploaded in line with the publishers’ terms and conditions and IS displayed in Open Search e.g. http://repository.leedsmet.ac.uk/main/view_record.php?identifier=696&SearchGroup=research)

---

*The only way I have been able to accommodate all of the information required for research (using EPrints software as a template) with the intraLibrary metadata schema (UK LOM) is by using multiple description fields as extensions – though these can be differentiated for display purposes, utilising a template to ensure consistent entry – e.g. http://repository.leedsmet.ac.uk/main/view_record.php?identifier=696&SearchGroup=research, they cannot easily be differentiated for search purposes.

---

Table 2 (cataloguing information to be completed by trained cataloguers)

Metadata element	Comments
Keyword	Controlled vocabulary utilising Library of Congress Authorities
Classification	Against (top two levels) of LCC

---

How metadata is related to the functionality of the Open Search interface

The current functionality of http://repository.leedsmet.ac.uk/main/index.php is as follows:

Function/field	Comments
Standard search	Comprises a simple search box that will search the entire metadata record – can manage simple Boolean operators (AND/OR) and perform a search to “Match exact text” (Boolean functionality needs reviewing)

---

Function/field	Comments
Advanced search (under development)	Comprises several metadata fields that can be cross referenced using simple Boolean logic (AND/OR – OR by default)
– Standard search	Should be self explanatory though early feedback from users suggests that it may not be!
– Title	Searches the “Title” metadata field only.
– ISBN/ISSN	In theory searches ISBN/ISSN – in practice searches all four description fields (see metadata Table 1 above)
– Author	In theory searches only the “Author” metadata (not yet implemented due to system bug – currently performing a standard search)
– Subject	Searches the “Subject” metadata field only.  NB.  Only one field – need to search multiple “Subject” fields.
– Type	Drop down list of vocab terms “Type of resource” (see metadata Table 1 above)
– Description/Abstract	In practice searches all four description fields (see metadata Table 1 above)
– DOI	Searches the “DOI” metadata field only
– Publisher	In theory searches only the “Publisher” metadata (not yet implemented due to system bug – currently performing a standard search)
– Format	Drop down list of all MIME types – in practice this will always be PDF for research – may be more appropriate for OER (see below)

---

Function	Comments
Browse	Currently two options to browse repository contents (research only)
Browse by Library of Congress category Browse by Leeds Met Faculty	Currently no way of knowing how many resources are available at a given level – requires further technical development to display “number of resources”

---

Open Educational Resources

I have so far been considering just research material but this, of course, is just one of the two main “types” of content that we need to manage with the Leeds Met repository.

Though infrastructure does not come within the remit of the UniCycle project, which is more focussed on the process around reuse of OER, an adequate search interface is something of a prerequisite; we have begun to integrate search functionality specific to OER into http://repository.leedsmet.ac.uk/main/index.php and are now able to differentiate between research and Open Educational Resources, for example, (using collection tokens), however, the metadata template for OER is quite different from research so ideally we will need a separate search form to reflect this.  Metadata for OER is currently based on the “old” Jorum template (i.e. not JorumOpen) and comprises the fields in Table 1a below:

Table 1a

Metadata element	Comments
Title	OER title
Description	Single instance only
Keyword	Uncontrolled/author produced keywords in line with ukoer guidelines (must include ukoer as keyword)
Contributor (Author)	System bug meant this could not be independently queried by SRU until recently – issue should now be resolved but not yet integrated with interface.
Contribution date	Automatically completed
Technical format	MIME media type – Comprising 70 technical formats – almost certainly don’t need them all for OER – are currently all exposed as list on advanced search form
Type of resource*	Terms from LOMv1.0: Diagram/ Exam/ Exercise/ Experiment/ Figure/ Graph/ Index/ Narrative Text/ Problem Statement/ Questionnaire/ Self Assessment/ Simulation/ Slide/ Table; Terms we have added to the vocabulary: Podcast/ Not Applicable/ Presentation/ Photograph/ Quiz/ Spreadsheet/ Tutorial/ Video/ Lecture/ Game/ Animation/ Assessment/ Audio/ Case Study/ Database/ Workbook N.B. As with research vocabulary, it is relatively straightforward to add new terms/modify existing terms though there may be work involved modifying records
Statement of Copyright and Restriction	As we are using Creative Commons licensing for OER we can probably dispense with this field – I will need to contact Intrallect to remove it from the template
Categorisation	OER are currently categorised against JACS (Joint Academic Coding System)

---

*The advanced search form currently only lists the research vocabulary in “Type” – this is an example of why we may need a separate form for advanced search of OER that displays the relevant vocabulary.

---

Display of OER

We also need to think about how OER records are displayed – currently this is a bit rough and ready, adapted from research results.  For example full text/external resource may not be as relevant for OER and  we may wish to display “Type of resource” here instead:

http://repository.leedsmet.ac.uk/main/search.php?q=ukoer&SearchSubmit.x=70&SearchSubmit.y=17&SearchSubmit=Search&SearchGroup[]=Open+Educational+Resources

---

Browse for OER?

In order to facilitate browse by subject heading (JACS) in the Open Search interface it will be necessary to integrate this functionality into http://repository.leedsmet.ac.uk/main/index.php in a similar way as has been implemented for research (same caveat applies in that further technical development is required to show numbers of resources in the browse tree.)