Really (not so) Simple Syndication

In theory RSS is simple, Really Simple, but the way that the technology is implemented by our underlying repository software (intraLibrary) and issues around how we have needed to integrate that software within our repository infrastructure in order to ensure appropriate Open Access has meant that, in reality, it has been anything but.

Broadly speaking, the issues are two fold:

• The fields exposed by the intralibrary RSS feed are limited to “Title” and “Description”.

• The URL exposed by the feed points to the public URL generated by intraLibrary (which would simply be the resource itself, either a file or URL i.e. without the context of the metadata record) whereas I need it to point to the Open Search metadata page.

I have been aware of these issues for some time but finding a resolution has been elevated in priority recently due to two separate, though similar, use-cases being explored by JorumOpen and the Xpert project at Nottingham University, that effectively seek to extend RSS from simply being a notification system to potentially also being used to harvest repository content (some are of the view that this not an appropriate use of the technology and that there are established technologies more suitable, specifically OAI-PMH and SWORD – see Intrallects’s Charles Duncan’s contribution to the discussion on Lorna Cambell’s blog).  The full discussion on Lorna’s post is certainly worth reading and also includes several contributions from Xpert’s Julian Tenney and Pat Lockley.

By now I have had an extended correspondence with Pat, initially prompted when I submitted the generic intraLibrary feed to be harvested by Xpert.  Preliminary feedback was that when Xpert tried to harvest our feed, it appeared that a randomly generated “key” was added to the URL, meaning they were seeing these urls as new resources, whereas they were actually duplicates (this was a consequence of the manner in which intraLibrary generates a publicURL each time a record is returned with a new machine generated “key” each time.)

As I described in a recent post, a little bit of Twitter serendipity and specifically input from Owen Stevens, subsequently led me to use Yahoo Pipes to redirect to the Open Search metadata page instead of the publicURL; Yahoo pipes also allows a pipe to be rendered as RSS and it occurred to me that this new feed could be resubmitted to Xpert for harvest.  Sure enough Pat confirmed that this new feed rendered from the pipe could indeed be harvested and that each URL was definitely unique (generated, of course, from resource unique IDs by http://repository.leedsmet.ac.uk/main/index.php – see http://repositorynews.wordpress.com/2009/11/09/leeds-met-repository-open-search-version-2-0/ for more info).  However, Pat also emphasised that it would be nice to find a generic way of harvesting without the pipe but for the time being I’m not sure we are in a position to implement such a solution; the work that we have done at Leeds Met on the Open Search interface and my personal obsession with redirecting RSS feeds is quite distinct from Intrallect’s primary commercial interest and so unlikely to be *officially* supported by the company (Note: As always Intrallect have been very supportive to me and certainly have proactively supported the development of our infrastructure throughout, it’s just that this isn’t a priority for them in the same way.)

As Owen has pointed out, all the Pipe does is take the ID which is part of the item’s GUID in the original RSS, and constructs a link to the metadata page which is put back into the RSS feed. You could obviously do this programatically (which might be a solution to Pat’s requirement of harvesting without the Pipe?) – it was just easier to throw together something quickly using Pipes and we simply don’t have the resources to explore another (programmatic) solution in detail.

Owen has also suggested that, as intraLibrary supports OAI-PMH, this would be the ‘supported’ mechanism for harvesting which really brings us full circle back to Charles’ argument referred to earlier in this post.

Note: On Lorna’s blog, Julian argues back at Charles that “OAI-PMH/SWORD etc are big technical barriers for many people who have resources to expose and that anyone can make a feed”; though in a subsequent comment, Pat does acknowledge that “it might be logical that if we are making a second RSS for harvesting we might use some other technology instead.”

All of which doesn’t get me very much further with my other RSS issue which is  the limited metadata with just <title> and <description> exposed by RSS and it was this that seemed to be more of an issue for harvest by JorumOpen with feedback from Gareth Waller confirming that the feed from intraLibrary “does not represent the metadata for the individual items (or the top level feed) in DC format (except of course date)” and in order for our feed to be processed with the code Gareth has implemented in JorumOpen , the feed would “need to contain DC metadata for each of the items and, more importantly, licence information”.  The feed from the Pipe, of course, still comprises just <title> and <description> and so would not be suitable for registering our OER content in JorumOpen.  (N.B. The limited metadata, of course, will surely also affect the quality of metadata harvested and searchable via Xpert?)

What would really make my life easier I think is for JorumOpen and Xpert to harvest my metadata using OAI-PMH (go on , you know you want to!) but for the time being I am just happy to have found a method to generate RSS feed that point to my static URLs at http://repository.leedsmet.ac.uk/main/index.php!

Using Yahoo Pipes to redirect to Open Search metadata page instead of intraLibrary public URL (and the power of Twitter)

First of all, a big thank you to Owen Stephens – @ostephens – who responded to my musing tweets on RSS by assembling a Pipe that “Rewrites Intralibrary RSS feed to use ‘link’ to metadata rather than object”; a great example of the power of Twitter for anyone who still thinks it’s an exercise in pointless self-revelation, full of trivial noise.  As Amber Thomas -@ambrouk – put it recently and as I also to tend towards, Owen exemplifies “whole person Tweeting” not restricting our interraction on Twitter to our professional sphere but filling it with more personal and sociable “noise” – the closest thing to a virtual office you will find.  I’ve never met Owen in real life but I shall certainly buy him a pint if our paths ever do cross!

As anyone who has passed by these parts before will know, we have been wrestling with intraLibrary for about two years now to develop a blended repository of Leeds Met’s research output (both Open Access full text and citation only) and Teaching & Learning material (both Open Educational Resources/material for federated access only) and we have spent a lot of time developing the IRISS SRU interface as a front end to provide appropriately differentiated Open Access to the different types of resources.

One of the simplest ways for a repository to alert users to new content is via RSS and it is very easy to generate a feed for pretty much any criteria in intraLibrary; I have generated several feeds for both research collections (by faculty) and for OER.  There are, however, two main issues with these feeds:

• The first problem is associated with the metadata template I have implemented for research and the lack of flexibility to customise which fields are exposed via RSS – I haven’t yet got a solution to this issue.
• the second problem, however, arises because the URL exposed by the feed points to the public URL generated by intraLibrary whereas I need it to point to the Open Search metadata page and this is where Yahoo Pipes can come in.

I don’t have much experience with Pipes but it is billed by Yahoo as “a free online service that lets you remix popular feed types and create data mashups using a visual editor. You can use Pipes to run your own web projects, or publish and share your own web services without ever having to write a line of code.”

Owen’s pipe has three components:

• “Fetch feed” which is simply the intraLibrary generated RSS feed
• “Regex” which applies a regular expression to an item attribute.  In this case it takes the components of the public URL in item.guid.content - oai:com\.intralibrary\.leedsmet:(.*) – and replaces it with the components to build the URL of the Open Search metadata page – http://repository.leedsmet.ac.uk/main/view_record.php?identifier=$1&SearchGroup=Open+Educational+Resources
• “Rename” which does what it says on the tin and simply renames or copies item atributes – in this case item.link becomes objectlink and item.guid.content becomes link

The Pipe Output can be subscribed to as RSS which gives us a feed that does indeed link to the Open Search metadata page rather than the intraLibrary public URL:

http://pipes.yahoo.com/pipes/pipe.run?_id=5c085e83cb144f9a1796558fa7d6d253&_render=rss

Simple when you know how!

N.B.  Some of these links actually DON’T work and I haven’t yet been able to figure out quite why.  As far as I can tell the affected resources were all uploaded as part of the Reproduce project and the links are to none-existent unique identifiers e.g.  http://repository.leedsmet.ac.uk/main/view_record.php?identifier=1432&SearchGroup=Open+Educational+Resources .  I think it may be because some of these records seem to have been ascribed 2 unique identifiers – this is an automatic process in intraLibrary and configured as uneditable so I’m not sure how it has happened.  However, they were originally uploaded by another user before the user profiles and metadata template for ukoer were fully configured and I may need to delete and re-upload as I have done already with http://repository.leedsmet.ac.uk/main/view_record.php?identifier=1673&SearchGroup=Open+Educational+Resources – I’m not sure how/when this will be updated in the feed and currently it’s still linking to the none-existent UID; it may take a little time to update.

Anyway, I’ve copied @ostephens Pipe – hope you don’t mind Owen, I couldn’t find any rights information(!) – and replaced the feed with one of my research feeds – Carnegie Faculty of Sport and Education – and modified the “Regex” module to redirect appropriately by replacing oai:com\.intralibrary\.leedsmet:(.*) with http://repository.leedsmet.ac.uk/main/view_record.php?identifier=$1&&SearchGroup=research.

N.B. As mentioned, there is an additional problem associated with the research metadata template and the lack of flexibility to customise which fields are exposed via RSS – the only way I have been able to accommodate all of the information required for research (using EPrints software as a template) with the intraLibrary metadata schema (UK LOM) is by using multiple description fields as extensions; ideally I would like the abstract to be exposed via RSS but this is in second description field whereas it is the bottommost description field that is exposed via RSS which generally contains whether the resource is refereed or not – this is not terribly relevant so I’ve added a Mapping to the “Rename” module to remove it which means the feed now exposes just the title which does indeed link to the Open Search metadata page:

http://pipes.yahoo.com/pipes/pipe.run?_id=286bbb1d8d30f65b54173b3b752fa4d9&_render=rss

A better beta – Open Search Version 2.0

After mounting pressure from the University, we have now added live links to http://repository.leedsmet.ac.uk/main/index.php from Library Online and the research web-pages but it is important that folk are aware that it is still very much a beta implementation – I have tried to summarise ongoing development requirements below (both for research material and Open Educational Resources) and I am very keen to receive user feedback to inform ongoing development; I also plan on doing some formal user evaluation in the New Year.

The two main issues we have faced throughout the development process is that intraLibrary is designed as a Learning Object repository requiring user authentication. This means we have needed to develop an appropriate metadata template for research material (mapping UK LOM onto Dublin Core) and develop an external, openly accessible search interface to query that metadata (via SRU) and display results appropriately.  Though there has been consultation throughout development I am still conscious that, *potentially*, there are issues outstanding that need to be more widely considered, especially as we move to establish a workflow involving more staff to populate the repository; obviously any changes to the metadata template will carry a more significant overhead the greater the number of records there are.

For research material, the workflow is currently configured in two stages and the approach we are currently considering would require me/designated administrator to complete the metadata in Table 1 and for trained cataloguers to complete the cataloguing information in Table 2:

---

Table 1 (primary metadata entry)

Metadata element	Comments
Faculty	6 Leeds Met faculties (& Thesis/dissertation)
Title	Article title
LOM identifier	Unique identifier applied automatically by intraLibrary
Digital Object Identifier	(if available) Utilising additional (editable) instance of LOM identifier field
Bibliographic metadata: Source title Journal volume Journal issue Source publication date Start page End page	It may not be necessary to complete all of these field depending on the “Type of resource”; they will all be required for a peer reviewed journal article but a book, for example, might only require “Source publication date” to be completed while a book chapter might require “Source title”; “Source publication date”; “Start page”; “End page” N.B.  A related issue that needs to be considered at this stage is that for records entered so far I have been inputting year only to the “Source publication date” field – however, we will need to differentiate records by census period; this perhaps could be achieved with a more specific date in the “Source pub date” field or we could explore using an additional field to record this information.
ISSN/ISBN*	Description field 1/4 – can’t really be differentiated from other description fields for search by SRU
Abstract*	Description field 2/4 – can’t really be differentiated from other description fields for search by SRU
Published/Not published*	Description field 3/4 – can’t really be differentiated from other description fields for search by SRU
Refereed/not refereed*	Description field 4/4 – can’t really be differentiated from other description fields for search by SRU
Author	System bug meant this could not be independently queried by SRU until recently – issue should now be resolved but not yet integrated with interface
Publisher	System bug meant this could not be independently queried by SRU until recently – issue should now be resolved but not yet integrated with interface
Contribution date	Automatically completed
Technical format	Always PDF for research
Type of resource	Scholarly text/ Book/ Book chapter/ Edited book/ Book review/ Miscellaneous conference item/ Conference contribution/ Journal item/ Electronic journal/ Electronic newspaper article/ Report/ Confidential report/ Submitted journal article/ Thesis or dissertation/ Working or discussion paper/ Lecture transcript NB. It is relatively straightforward to add terms/modify existing terms though there may be work involved modifying records. For example, it has been suggested that the vocabulary term “Journal item” should be changed to “Journal article” and while this is relatively straightforward, there is no way of making a global change without reviewing each record individually – currently 218 items.
Statement of Copyright and Restriction	Where a record is citation only (i.e. no full text available) I am using this field to record information from SHERPA/RoMEO to indicate whether I should pursue the full text (for my reference only – i.e. not displayed in Open Search interface).  When there is a full text available this field will generally record that it has been uploaded in line with the publishers’ terms and conditions and IS displayed in Open Search e.g. http://repository.leedsmet.ac.uk/main/view_record.php?identifier=696&SearchGroup=research)

---

*The only way I have been able to accommodate all of the information required for research (using EPrints software as a template) with the intraLibrary metadata schema (UK LOM) is by using multiple description fields as extensions – though these can be differentiated for display purposes, utilising a template to ensure consistent entry – e.g. http://repository.leedsmet.ac.uk/main/view_record.php?identifier=696&SearchGroup=research, they cannot easily be differentiated for search purposes.

---

Table 2 (cataloguing information to be completed by trained cataloguers)

Metadata element	Comments
Keyword	Controlled vocabulary utilising Library of Congress Authorities
Classification	Against (top two levels) of LCC

---

How metadata is related to the functionality of the Open Search interface

The current functionality of http://repository.leedsmet.ac.uk/main/index.php is as follows:

Function/field	Comments
Standard search	Comprises a simple search box that will search the entire metadata record – can manage simple Boolean operators (AND/OR) and perform a search to “Match exact text” (Boolean functionality needs reviewing)

---

Function/field	Comments
Advanced search (under development)	Comprises several metadata fields that can be cross referenced using simple Boolean logic (AND/OR – OR by default)
- Standard search	Should be self explanatory though early feedback from users suggests that it may not be!
- Title	Searches the “Title” metadata field only.
- ISBN/ISSN	In theory searches ISBN/ISSN – in practice searches all four description fields (see metadata Table 1 above)
- Author	In theory searches only the “Author” metadata (not yet implemented due to system bug – currently performing a standard search)
- Subject	Searches the “Subject” metadata field only.  NB.  Only one field – need to search multiple “Subject” fields.
- Type	Drop down list of vocab terms “Type of resource” (see metadata Table 1 above)
- Description/Abstract	In practice searches all four description fields (see metadata Table 1 above)
- DOI	Searches the “DOI” metadata field only
- Publisher	In theory searches only the “Publisher” metadata (not yet implemented due to system bug – currently performing a standard search)
- Format	Drop down list of all MIME types – in practice this will always be PDF for research – may be more appropriate for OER (see below)

---

Function	Comments
Browse	Currently two options to browse repository contents (research only)
Browse by Library of Congress category Browse by Leeds Met Faculty	Currently no way of knowing how many resources are available at a given level – requires further technical development to display “number of resources”

---

Open Educational Resources

I have so far been considering just research material but this, of course, is just one of the two main “types” of content that we need to manage with the Leeds Met repository.

Though infrastructure does not come within the remit of the UniCycle project, which is more focussed on the process around reuse of OER, an adequate search interface is something of a prerequisite; we have begun to integrate search functionality specific to OER into http://repository.leedsmet.ac.uk/main/index.php and are now able to differentiate between research and Open Educational Resources, for example, (using collection tokens), however, the metadata template for OER is quite different from research so ideally we will need a separate search form to reflect this.  Metadata for OER is currently based on the “old” Jorum template (i.e. not JorumOpen) and comprises the fields in Table 1a below:

Table 1a

Metadata element	Comments
Title	OER title
Description	Single instance only
Keyword	Uncontrolled/author produced keywords in line with ukoer guidelines (must include ukoer as keyword)
Contributor (Author)	System bug meant this could not be independently queried by SRU until recently – issue should now be resolved but not yet integrated with interface.
Contribution date	Automatically completed
Technical format	MIME media type – Comprising 70 technical formats – almost certainly don’t need them all for OER – are currently all exposed as list on advanced search form
Type of resource*	Terms from LOMv1.0: Diagram/ Exam/ Exercise/ Experiment/ Figure/ Graph/ Index/ Narrative Text/ Problem Statement/ Questionnaire/ Self Assessment/ Simulation/ Slide/ Table; Terms we have added to the vocabulary: Podcast/ Not Applicable/ Presentation/ Photograph/ Quiz/ Spreadsheet/ Tutorial/ Video/ Lecture/ Game/ Animation/ Assessment/ Audio/ Case Study/ Database/ Workbook N.B. As with research vocabulary, it is relatively straightforward to add new terms/modify existing terms though there may be work involved modifying records
Statement of Copyright and Restriction	As we are using Creative Commons licensing for OER we can probably dispense with this field – I will need to contact Intrallect to remove it from the template
Categorisation	OER are currently categorised against JACS (Joint Academic Coding System)

---

*The advanced search form currently only lists the research vocabulary in “Type” – this is an example of why we may need a separate form for advanced search of OER that displays the relevant vocabulary.

---

Display of OER

We also need to think about how OER records are displayed – currently this is a bit rough and ready, adapted from research results.  For example full text/external resource may not be as relevant for OER and  we may wish to display “Type of resource” here instead:

http://repository.leedsmet.ac.uk/main/search.php?q=ukoer&SearchSubmit.x=70&SearchSubmit.y=17&SearchSubmit=Search&SearchGroup[]=Open+Educational+Resources

---

Browse for OER?

In order to facilitate browse by subject heading (JACS) in the Open Search interface it will be necessary to integrate this functionality into http://repository.leedsmet.ac.uk/main/index.php in a similar way as has been implemented for research (same caveat applies in that further technical development is required to show numbers of resources in the browse tree.)

Musing about metadata for OER

(Disclaimer: I am not a qualified librarian or cataloguer.)

Thus far I’ve only really considered, in detail,  the minutiae of research specific metadata (a consequence of the ongoing project to repurpose an LO repository as an OA archive of research).  However, this morning I uploaded my first learning object to intraLibrary that has been specifically designated as an OER for the Unicycle project.  The resource itself isn’t terribly exciting in that it is just a Word.doc but it makes sense to start with a relatively simple object than a more exotic file type or package.

The metadata template I’m using is taken directly from JORUM without any modification and is consequently (currently at least) based on IEEE LOM though it’s worth restating that we’ve recently learned that JORUM will be using DSpace to serve OER rather than intraLibrary so that service’s template is almost certain to change considerably in the very near future.

Mandatory fields:

Title
Description(s)
Keyword(s)
Contributor(s): Author(s)
Contribution Date
Payment Required (yes/no)
Subject to Copyright (yes/no)
Statement of Copyright and Restriction

Recommended Fields:

Version
Status
IMS LRM Metadata Identifier
Role of Metadata Contributor
Metadata Contributor(s)
Date of Metadata Contribution
Language of Metadata Record
Technical Format
Location of Resource
Language of Intended User
Type of Resource
Intended for Use In
Level of Difficulty

Optional Fields:

Description of Contribution Date
Module Code
Module Title
IMS LRM Metadata Identifier
Catalogue
Description of Metadata Contribution Date
Format of Metadata Record
Size Resource in Bytes
Type of Delivery System
Name of Delivery System
Minimum Version of Delivery System
Maximum Version of Delivery System
How to Install This Resource
Special Requirements for Use
Duration of Media Resource: Playing Time
Type of Interactivity
Educational Purpose Description
Language of Intended User
Level of Interactivity
Semantic Density
Intended for Use By
Completion Time
CLA Reporting Data

In addition, the resource is classified against JACS (Joint Academic Coding System) – again emulating the current (intraLibrary) JORUM configuration and can be ascribed one of the six Creative Commons attribution licences.

So off I went.

Title is straightforward, and Description – though this field does require some thought to make it useful – but what about Keyword(s)? My first thought is that we need some sort of controlled vocabulary – for research material I am using Library of Congress Authorities which is well established practice for this type of material but perhaps less appropriate for learning objects; one could rely on author produced keywords but these are perhaps not easily generated as formal metadata and will tend to lack consistency (social tagging is perhaps a slightly different issue and intraLibrary does have a facility for ascribing user-generated tags after the resource has been published.)

I posted my idle musings to Twitter and was immediately called to account by @philbarker and @LornaMCampbell who both asked what specific aspects of the learning object the controlled vocab would need to describe. @ambrouk acknowledged it was a big question and suggested two separate criteria: a) subject classification b) everything else author might tell you about / user says they want?

Input also came from @lynncorrigan and @KavuBob who pointed me to the HILT (High Level Thesaurus) project at Strathclyde – http://hilt.cdlr.strath.ac.uk/hilt4/demonstrators.html

I haven’t yet looked closely at this project but from the Twitter response it’s clearly an area that needs further research so this is a quick post to further the discussion…I’ll come back when I’ve explored the issue in more detail and all perspectives in the meantime gratefully received.

Some final unstructured thoughts at this stage:

• The need for a lightweight metadata template – which I guess the mandatory subset is though even this requires Keyword(s) AND classification against JACS…AND social tags can also be applied.  To what extent the recommended/optional fields need to be completed depends to some extent on the type of resource though I suspect that some of these fields will actually need to be mandatory for certain resource types – Duration of Media Resource: Playing time for example
• How the metadata relates to resource discovery on the modern web – metadata is ultimately to aid resource discovery (and nail down copyright and licencing).  Where will our OERs be primarily discovered from?  A Leeds Met interface/a JORUM interface/Google/All of the above/somewhere else?
• Workflows – who uploads resources and applies metadata?  The creator/repository administrator?  Does there perhaps need to be a multiple stage workflow utilising something like the recently released JORUM quick deposit tool (ideally based on SWORD) whereby metadata is subsequently enriched by trained staff?

Repositories for research and teaching/learning material: The debate continues at #rpmeet

Last week I attended the JISC Repository and Preservation end of programme meeting in Birmingham. I recall being very nervous at my first JISC event in November 2007 but feel much more at ease now and enjoyed the event immensely; the programme has certainly been successful in fostering a sense of community though it’s an unusual social experience to meet people face to face, often for the very first time, when one feels you already know them from reading their blog and following them on Twitter.

During one of the breakout sessions on the first day I made a bee-line for a discussion about repositories for learning and teaching materials – as opposed to OA research repositories. I use the word “opposed” advisedly as there is certainly some strong sentiment around the issue, particularly with respect to using a common software platform. As a representative of a project that is adapting a learning object repository to also serve as an effective Open Access research repository I’m finding it a little difficult to understand the vehemence of some of this opposition, though I would be the first to acknowledge a steep learning curve and recognise that we have required extensive development, not of intraLibrary itself perhaps, but of an appropriate web infrastructure surrounding it. And yes, we would certainly have been able to implement a functioning OA research repository more quickly using EPrints or DSpace however, from the outset, it was vital that our repository had the the capacity to fulfil its broader potential – in the words of Clifford Lynch “[A] mature and fully realised institutional repository will contain the intellectual works of faculty and students – both research and teaching materials – and also documentation of the activities of the institution itself in the form of records of events and performance and of the ongoing intellectual life of the institution.”  [Lynch, Clifford. A “Institutional Repositories: Essential Infrastructure for Scholarship in the Digital Age” ARL Bimonthly Report 226 (2003).]

It’s also important to be pragmatic.  Historically, Leeds Metropolitan University is a polytechnic that gained chartered university status in 1992; its heritage is very much in teaching and learning rather than research with, arguably, a more vocational than academic flavour.  In recent years, the research profile has steadily increased, culminating in unprecedented success in the 2008 RAE and the university is naturally keen to capitalise on this success, enhance its research profile further whilst also continuing to emphasise its student focussed teaching and learning credentials. The implementation of an integrated repository to support both research outputs and learning objects reflects this dual focus.  Clifford Lynch’s article suggests that the concept of a central system to manage disparate resources in this way has been implicit within the sector for some years, however, the technology has tended to focus on Open Access to research, with the two most widely used software platforms being EPrints, developed at the University of Southampton in 2000, and DSpace, developed at MIT in 2002; early versions of both platforms were primarily designed to manage text based resources (though subsequent versions of EPrints and DSpace can manage a wide range of digital file formats.)  

NB.  In an extended discussion on this issue on JISC-REPOSITORIES (archive here) RepositoryMan Les Carr of EPrints refers to the fact that he still comes across the firmly held (and spurious) belief that because EPrints is used for Open Access it can’t be used for multimedia files or scientific data.

The session was chaired by Amber Thomas of JISC and I asked a somewhat blunt, perhaps naive, question about JISC’s perspectives on combined repositories of research and teaching materials.  Amber suggested that JISC have been deliberately neutral on the issue which is also perhaps emphasised by the diagramatic representation of the programme structure reproduced above.  

Some of the commentators last Wednesday were adamant that though it may well be possible to manage different types of resources with a single system it was far from desirable with one colleague making the pithy analogous observation that you can write letters in Excel but that doesn’t make it right.  Phil Barker of CETIS was also at the discussion and in a recent blog post on the “question of whether research outputs and learning materials should stored in the same repository” is “inclined to think the answer is no, the purpose of the repository is different, a learning material isn’t an output, sharing means something different for the two resource types.”  Phil goes on to say that ” If you think a repository is a database and a bit-store then you may come to a different conclusion, but I think a repository is a service offered to people and your choice of starting point in offering that service will affect how easy your journey is.”  (Full post here)

I’d certainly concede that our journey hasn’t been an easy one and I also agree that a repository is a service offered to people and with our repository start-up, and also Streamline and PERSoNA, that is certainly the approach we have tried to take; with intraLibrary and the SRU interface we now have an incipient infrastructure to manage both research material and learning objects; the discrete types of material can be managed entirely separately, however, there is also potential for the ongoing development of a holistic approach to the management of the full range of digital resources produced by a modern university and as we develop our infrastructure further I hope we can utilise appropriate web-technology around a central management system (intraLibrary) to achieve decentralised resource discovery – through appropriate interfaces, widgets and environments – the VLE for example.

Then of course there is the small matter of persuading academics to part with their resources, not to mention IPR, copyright and quality control issues…

Open Access to research is an evolving paradigm and represents a considerable shift in the established academic publishing process; Open Access to a broader range of educational resources still more so. Any paradigm shift is likely to take time to evolve and Open Access, to research and other materials, is no exception, especially given that academia, perhaps, tends to subscribe rather strongly to established tradition!

JISC’s current OER programme should go some way to addressing many of these issues but infrastructure is the foundation. The perfect system almost certainly doesn’t exist and it’s surely important to be pragmatic when implementing and developing appropriate system. Here’s to ongoing discussion, debate and development.

Google indexing and SEO

It is crucial that both the Open Access full text research content of the repository and metadata records of citation material are fully indexed by Google (and other search engines); in the future it is also likely to be required for other Open Educational Resources (learning objects). However, site:http://repository-intralibrary.leedsmet.ac.uk/ currently returns just 4 results (in addition to the Login page itself) and it is a bit of a mystery how these 4 are actually being picked up when the majority of records are not.

In intraLibrary, for a given collection, the administrator may choose to:

• Allow published content in this collection to be searched by external systems

This effectively means SRU (Search and Retrieve by URL) a standard search protocol utilizing CQL (Common Query Language).

• Allow published records in this collection to be harvested by external systems

This effectively means harvest by OAI-PMH

XML Sitemaps

Intrallect have suggested that it is necessary to implement an XML sitemap to ensure that content is properly crawled by Google. Until 2008, Google did support sitemaps using OAI-PMH but have since withdrawn this and now support only the standard XML format. Intrallect have therefore developed a software tool that converts OAI-PMH output to an appropriate XML format. A sitemap has been generated and registered using Google’s webmaster tools but currently is registering a series of errors that indicate “This URL is not allowed for a Sitemap at this location”; 9 errors are listed from the very first URL and which are sequential; it seems that the crawl does not go any further and none of the 100+ URLs in the sitemap have been successfully recognised. Two possible reasons have been suggested for this:

• All of the URLs in the sitemap are external; it may be that Google does not permit URLs outside the mapped domain.
• There is a problem with the XML itself

Sitemap here: http://repository-intralibrary.leedsmet.ac.uk/sitemap/Sitemap.xml

Sitemaps using RSS

It is also possible to submit a sitemap based on RSS, however, this approach has not been any more successful as the Open URL/virtual file paths generated by intraLibrary are inaccessible to Google resulting in the following warning:

URLs not followed
When we tested a sample of URLs from your Sitemap, we found that some URLs redirect to other locations. We recommend that your Sitemap contain URLs that point to the final destination (the redirect target) instead of redirecting to another URL.

Google and SRU

Though SRU does not facilitate indexing by Google per se, the integration of the SRU Open Search interface may provide a potential solution. site:http://repository.leedsmet.ac.uk/ currently returns 247 records; largely these appear to represent Googlebot following the various browse links (many of which themselves return no results where there is no content to find!) In addition, Googlebot appears to be following hyperlinked author names, publisher and subject(s) in the individual metadata records:

The third of these “The Repository search for Morton, Veronica” links to the two metadata records associated with that name as though it had simply been entered into http://repository.leedsmet.ac.uk/ as a search term:

http://repository.leedsmet.ac.uk/main/search.php?q=Morton%2C+Veronica+

Presumably these records were initially indexed via the appropriate links on the browse interface – http://repository.leedsmet.ac.uk/main/browse.php – Faculty of Health and R – Medicine and then re-indexed via the hyperlinks embedded in the metadata records. It is interesting to note that, though Morton, Veronica only has two records associated with her name, this record appears relatively high – at the top of the second page – and this is probably because there so many other authors also associated with these papers; all of these names are hyperlinked giving over 21 separate indexable links.

It seems that we might need to formalise the structure of the SRU to ensure it is optimised for Google; possibly with some sort of SRU sitemap. For example, if we could generate a page that linked to all the individual metadata records in the repository and optimise this page to be crawled by search engine spiders (doesn’t need to be human readable; could be XML) which could then follow the links to the associated metadata.

It also seems to me that Search Engine Optimisation will need to comprise appropriate customisation of the SRU interface; for example, we want to facilitate browse by author which, in turn, will provide indexable links for Googlebot.

Full text indexing

There is also the issue of indexing full text. As already mentioned, Google does not follow the Open URL/virtual file paths generated by intraLibrary and all the results from site:http://repository.leedsmet.ac.uk/ are search results. Potentially this is a benefit in as much as people are less likely to bypass the metadata record and go directly to the PDF but we do also want to facilitate full text indexing. We may have to wait for Intrallect on this who have assured us they are looking into facilitating full text indexing – probably via intraLibrary itself rather than the SRU.