Friday Mash (with a little help)

This might be messy – just need to brain dump to try and figure this out. Thanks again to the Godfather of Mashed Libraries @ostephens for his help!

The problem(s):

intraLibrary RSS feeds point to resource “in the wild” rather than the record on Open Search. Like this:

http://repository-intralibrary.leedsmet.ac.uk/IntraLibrary-RSS?rss_feed_id=6a6176612e7574696c2e52616e646f6d4031393166376364&rss_2.0.xml

Owen helped me put together a simple pipe that took this feed and used regex to replace an identifier from the record and redirect to the Open Search URL (which is built from this identifier. Like this:

http://pipes.yahoo.com/pipes/pipe.edit?_id=3586ad8141869b6978d8dbef538ac01f

So far so good…however, when I originally defined my Application Profile for research in intraLibrary I used multiple instances of <lom:description> with the first holdong ISSN (frustratingly missing from intraLibraries Bib extensions) and the Abstract held in a second instance of <lom:description> meaning it isn’t exposed in an RSS feed.

So…I thought that if I used an SRU query instead as a pipe input there would be a lot more data to play with in Pipes and hopefully I would be able to build a better RSS feed – including author and abstract.

After some initial problems with Pipes taking an SRU input, Owen responded to my plea for help with this pipe that extracts title and abstract from the SRU by defining the path through the XML to the relevant fields and mapping them to title and description:

http://pipes.yahoo.com/pipes/pipe.edit?_id=2054b55b81b5b96236b6870b03bef85c

However, I still need to figure out how to link the title to the respective record on Open Search. There is a link in the XML but this is no good as, once again, it points to the resource in the wild rather than the record on Open Search…somehow I need to use the identifier to build a link to the respective record on Open Search.

<lom:entry>oai:com.intralibrary.leedsmet:3352</lom:entry>

http://repository.leedsmet.ac.uk/main/view_record.php?identifier=3352&SearchGroup=research

And frankly now I’m a bit stumped again – the regex function from the first pipe presumably needs to be in there somewhere…first vague attempt (doesn’t actually return any output – but this is a brain dump!):

http://pipes.yahoo.com/pipes/pipe.edit?_id=00bc8c1cd13fc852d2fb8d0da7defdd4

Ongoing RSS issue

To cut a long story short, I run RSS feeds from intraLibrary through Yahoo Pipes to redirect to the metadata page on Open Search – http://repository.leedsmet.ac.uk/main/index.php

However, due to the way I initially mapped intraLibrary LOM, the abstract is held in a second instance of the description field so my RSS feeds are just title (I’ve got ISSN in the first which I hide)…sure there must be a way to produce an RSS feed from an SRU query including this field using Yahoo Pipes…this is the SRU query:

http://repository-intralibrary.leedsmet.ac.uk/IntraLibrary-SRU?recordSchema=lom&operation=searchRetrieve&version=1.1&maximumRecords=10&startRecord=1&query=rec.collectionIdentifier=Carnegie&x-info-2-auth1.0-authenticationToken=research

Anybody help?

End of a pipe dream?

I’ve been enjoying exploring Yahoo Pipes and have now managed to generate a feed for OER that also incorporates the author vCard; I’m sure it must also be possible to extract other data including rights information (though I’m not quite sure how!):

http://pipes.yahoo.com/mrnick/build_rss_from_xml

(vCard not visible in the Pipe but is displayed if the Pipe is rendered as RSS)

It also occurred to me that it may be possible to use a similar method to extract the abstract for research RSS feeds (which is in the second description field) and I’ve been grappling with Pipes to this end but am struggling at the moment – tantalisingly I am able to construct a “Path to item list” that returns an individual abstract but for the life of me, can’t figure out how to return abstracts for all records – SRW:records.SRW:record.1.SRW:recordData.lom:lom.lom:general.lom:description.1.lom:string.
content

I’ll certainly continue to use Pipes as it’s a very powerful tool, especially for a non-programmer like me, however, for the time being this mini-project is postponed. Nothing is ever wasted though and I’m sure what I’ve learned will come in useful somewhere down the line….

Pipe dream

This afternoon I’ve been grappling with Yahoo Pipes, trying to generate a feed that incorporates metadata from an SRU query.  I’ve made a modicum of progress and, as I’m now a bit stuck, this is just a quick post to document that modicum before it all leaks away over the weekend.

The Pipe module that seems most appropriate is “Fetch Data”: “This module retrieves any XML, JSON, iCal or KML file and tries to extract a list of elements using the provided path parameter.”

I input an appropriate URL to query “ukoer” which will return all OERs uploaded as part of UniCycle:

http://repository-intralibrary.leedsmet.ac.uk/IntraLibrary-SRU?recordSchema=lom&maximumRecords=50&operation=searchRetrieve&version=1.1&query=ukoer&SearchSubmit.x=18&SearchSubmit.y=24&x-info-2-auth1.0-authenticationToken=oer

And read the instructions here which describe how to use the “Path to item list” field in order to extract just a portion of the data by listing the nested XML elements, separating each with a dot (“.”)

After a bit of trial and error I was able to zero in on the “Description” field for the first record by entering SRW:records.SRW:record.0.SRW:recordData.lom:lom.lom:general.lom:description.lom:string.content into “path to item list”

However, when I attempt to run the pipe, nothing is returned  – though the correct field IS displayed correctly in the debugger panel:

Also, what I would like to do of course, is to return the “Description” field for ALL records but I have no idea how to achieve this…I’ll have another look when I’m nice and fresh on Monday morning!

Really (not so) Simple Syndication

In theory RSS is simple, Really Simple, but the way that the technology is implemented by our underlying repository software (intraLibrary) and issues around how we have needed to integrate that software within our repository infrastructure in order to ensure appropriate Open Access has meant that, in reality, it has been anything but.

Broadly speaking, the issues are two fold:

• The fields exposed by the intralibrary RSS feed are limited to “Title” and “Description”.

• The URL exposed by the feed points to the public URL generated by intraLibrary (which would simply be the resource itself, either a file or URL i.e. without the context of the metadata record) whereas I need it to point to the Open Search metadata page.

I have been aware of these issues for some time but finding a resolution has been elevated in priority recently due to two separate, though similar, use-cases being explored by JorumOpen and the Xpert project at Nottingham University, that effectively seek to extend RSS from simply being a notification system to potentially also being used to harvest repository content (some are of the view that this not an appropriate use of the technology and that there are established technologies more suitable, specifically OAI-PMH and SWORD – see Intrallects’s Charles Duncan’s contribution to the discussion on Lorna Cambell’s blog).  The full discussion on Lorna’s post is certainly worth reading and also includes several contributions from Xpert’s Julian Tenney and Pat Lockley.

By now I have had an extended correspondence with Pat, initially prompted when I submitted the generic intraLibrary feed to be harvested by Xpert.  Preliminary feedback was that when Xpert tried to harvest our feed, it appeared that a randomly generated “key” was added to the URL, meaning they were seeing these urls as new resources, whereas they were actually duplicates (this was a consequence of the manner in which intraLibrary generates a publicURL each time a record is returned with a new machine generated “key” each time.)

As I described in a recent post, a little bit of Twitter serendipity and specifically input from Owen Stevens, subsequently led me to use Yahoo Pipes to redirect to the Open Search metadata page instead of the publicURL; Yahoo pipes also allows a pipe to be rendered as RSS and it occurred to me that this new feed could be resubmitted to Xpert for harvest.  Sure enough Pat confirmed that this new feed rendered from the pipe could indeed be harvested and that each URL was definitely unique (generated, of course, from resource unique IDs by http://repository.leedsmet.ac.uk/main/index.php – see https://repositorynews.wordpress.com/2009/11/09/leeds-met-repository-open-search-version-2-0/ for more info).  However, Pat also emphasised that it would be nice to find a generic way of harvesting without the pipe but for the time being I’m not sure we are in a position to implement such a solution; the work that we have done at Leeds Met on the Open Search interface and my personal obsession with redirecting RSS feeds is quite distinct from Intrallect’s primary commercial interest and so unlikely to be *officially* supported by the company (Note: As always Intrallect have been very supportive to me and certainly have proactively supported the development of our infrastructure throughout, it’s just that this isn’t a priority for them in the same way.)

As Owen has pointed out, all the Pipe does is take the ID which is part of the item’s GUID in the original RSS, and constructs a link to the metadata page which is put back into the RSS feed. You could obviously do this programatically (which might be a solution to Pat’s requirement of harvesting without the Pipe?) – it was just easier to throw together something quickly using Pipes and we simply don’t have the resources to explore another (programmatic) solution in detail.

Owen has also suggested that, as intraLibrary supports OAI-PMH, this would be the ‘supported’ mechanism for harvesting which really brings us full circle back to Charles’ argument referred to earlier in this post.

Note: On Lorna’s blog, Julian argues back at Charles that “OAI-PMH/SWORD etc are big technical barriers for many people who have resources to expose and that anyone can make a feed”; though in a subsequent comment, Pat does acknowledge that “it might be logical that if we are making a second RSS for harvesting we might use some other technology instead.”

All of which doesn’t get me very much further with my other RSS issue which is  the limited metadata with just <title> and <description> exposed by RSS and it was this that seemed to be more of an issue for harvest by JorumOpen with feedback from Gareth Waller confirming that the feed from intraLibrary “does not represent the metadata for the individual items (or the top level feed) in DC format (except of course date)” and in order for our feed to be processed with the code Gareth has implemented in JorumOpen , the feed would “need to contain DC metadata for each of the items and, more importantly, licence information”.  The feed from the Pipe, of course, still comprises just <title> and <description> and so would not be suitable for registering our OER content in JorumOpen.  (N.B. The limited metadata, of course, will surely also affect the quality of metadata harvested and searchable via Xpert?)

What would really make my life easier I think is for JorumOpen and Xpert to harvest my metadata using OAI-PMH (go on , you know you want to!) but for the time being I am just happy to have found a method to generate RSS feed that point to my static URLs at http://repository.leedsmet.ac.uk/main/index.php!

Using Yahoo Pipes to redirect to Open Search metadata page instead of intraLibrary public URL (and the power of Twitter)

First of all, a big thank you to Owen Stephens – @ostephens – who responded to my musing tweets on RSS by assembling a Pipe that “Rewrites Intralibrary RSS feed to use ‘link’ to metadata rather than object”; a great example of the power of Twitter for anyone who still thinks it’s an exercise in pointless self-revelation, full of trivial noise.  As Amber Thomas –@ambrouk – put it recently and as I also to tend towards, Owen exemplifies “whole person Tweeting” not restricting our interraction on Twitter to our professional sphere but filling it with more personal and sociable “noise” – the closest thing to a virtual office you will find.  I’ve never met Owen in real life but I shall certainly buy him a pint if our paths ever do cross!

As anyone who has passed by these parts before will know, we have been wrestling with intraLibrary for about two years now to develop a blended repository of Leeds Met’s research output (both Open Access full text and citation only) and Teaching & Learning material (both Open Educational Resources/material for federated access only) and we have spent a lot of time developing the IRISS SRU interface as a front end to provide appropriately differentiated Open Access to the different types of resources.

One of the simplest ways for a repository to alert users to new content is via RSS and it is very easy to generate a feed for pretty much any criteria in intraLibrary; I have generated several feeds for both research collections (by faculty) and for OER.  There are, however, two main issues with these feeds:

• The first problem is associated with the metadata template I have implemented for research and the lack of flexibility to customise which fields are exposed via RSS – I haven’t yet got a solution to this issue.
• the second problem, however, arises because the URL exposed by the feed points to the public URL generated by intraLibrary whereas I need it to point to the Open Search metadata page and this is where Yahoo Pipes can come in.

I don’t have much experience with Pipes but it is billed by Yahoo as “a free online service that lets you remix popular feed types and create data mashups using a visual editor. You can use Pipes to run your own web projects, or publish and share your own web services without ever having to write a line of code.”

Owen’s pipe has three components:

• “Fetch feed” which is simply the intraLibrary generated RSS feed
• “Regex” which applies a regular expression to an item attribute.  In this case it takes the components of the public URL in item.guid.content – oai:com\.intralibrary\.leedsmet:(.*) – and replaces it with the components to build the URL of the Open Search metadata page – http://repository.leedsmet.ac.uk/main/view_record.php?identifier=$1&SearchGroup=Open+Educational+Resources
• “Rename” which does what it says on the tin and simply renames or copies item atributes – in this case item.link becomes objectlink and item.guid.content becomes link

The Pipe Output can be subscribed to as RSS which gives us a feed that does indeed link to the Open Search metadata page rather than the intraLibrary public URL:

http://pipes.yahoo.com/pipes/pipe.run?_id=5c085e83cb144f9a1796558fa7d6d253&_render=rss

Simple when you know how!

N.B.  Some of these links actually DON’T work and I haven’t yet been able to figure out quite why.  As far as I can tell the affected resources were all uploaded as part of the Reproduce project and the links are to none-existent unique identifiers e.g.  http://repository.leedsmet.ac.uk/main/view_record.php?identifier=1432&SearchGroup=Open+Educational+Resources .  I think it may be because some of these records seem to have been ascribed 2 unique identifiers – this is an automatic process in intraLibrary and configured as uneditable so I’m not sure how it has happened.  However, they were originally uploaded by another user before the user profiles and metadata template for ukoer were fully configured and I may need to delete and re-upload as I have done already with http://repository.leedsmet.ac.uk/main/view_record.php?identifier=1673&SearchGroup=Open+Educational+Resources – I’m not sure how/when this will be updated in the feed and currently it’s still linking to the none-existent UID; it may take a little time to update.

Anyway, I’ve copied @ostephens Pipe – hope you don’t mind Owen, I couldn’t find any rights information(!) – and replaced the feed with one of my research feeds – Carnegie Faculty of Sport and Education – and modified the “Regex” module to redirect appropriately by replacing oai:com\.intralibrary\.leedsmet:(.*) with http://repository.leedsmet.ac.uk/main/view_record.php?identifier=$1&&SearchGroup=research.

N.B. As mentioned, there is an additional problem associated with the research metadata template and the lack of flexibility to customise which fields are exposed via RSS – the only way I have been able to accommodate all of the information required for research (using EPrints software as a template) with the intraLibrary metadata schema (UK LOM) is by using multiple description fields as extensions; ideally I would like the abstract to be exposed via RSS but this is in second description field whereas it is the bottommost description field that is exposed via RSS which generally contains whether the resource is refereed or not – this is not terribly relevant so I’ve added a Mapping to the “Rename” module to remove it which means the feed now exposes just the title which does indeed link to the Open Search metadata page:

http://pipes.yahoo.com/pipes/pipe.run?_id=286bbb1d8d30f65b54173b3b752fa4d9&_render=rss