Everybody's Libraries

January 28, 2009

Open catalog APIs and data: ALA presentation notes posted

Filed under: architecture,discovery,libraries,open access,sharing — John Mark Ockerbloom @ 3:48 pm

I’ve now posted my materials for the two panels I participated in at ALA Midwinter.

I have slides  available for “Opening the ILS for Discovery: The Digital Library Federation’s ILS-Discovery Interface Recommendations“, a presentation for LITA’s Next Generation Catalog interest group, where I gave an overview of the recommendations and their use.   At the same session, Beth Jefferson of BiblioCommons talked about some of the social and legal issues of sharing user content in library catalogs and other discovery applications.

And I have the slides and remarks I prepared for  “Open Records, Open Possibilities“, a presentation for the ALCTS panel on shared bibliographic records and the future of WorldCat.  In that one, I argue for more open access to bibliographic records, showing some of the benefits and sustainability strategies of open access models.

Karen Calhoun has also posted the slides from her presentation at that panel.  Peter Murray also presented; I haven’t yet found his slides online, but he’s blogging about what he said.  The fourth panelist, Brian Schottlaender, didn’t present slides, but instead gave thoughtful summaries and follow-on questions to some of the points the rest of us made.  From the audience, Norman Oder of Library Journal took notes and then wrote a useful report on the session.

I’d like to thank the organizers of these sessions, Sharon Shafer and Charles Wilt, for inviting me to speak, and my co-presenters for sharing their ideas and viewpoints.

December 9, 2008

Revised ILS-Discovery interface recommendation released

Filed under: architecture,discovery,libraries — John Mark Ockerbloom @ 3:40 pm

I’ve just sent the following announcement out to the ILS-Discovery Interface Google Group:

The Digital Library Federation’s ILS-DI task group has officially released revision 1.1 of their recommendation for standard interfaces for integrating the data and services of the Integrated Library System (ILS) with new applications supporting user discovery.

Our initial official release (“revision 1.0”) was made in June, and included a recommendation of a basic level of interoperability (the Basic Discovery Interfaces, or “Level 1” interoperability) that was agreed to by many ILS vendors in the “Berkeley Accord“.

In August, the DLF convened an implementor’s meeting in Berkeley that was attended by a number of developers and vendors of ILS and discovery software.  In the meeting, we agreed to make certain changes to clarify the requirements of the basic level of compliance, and to make them more useful for discovery applications.  A revised draft that included these changes was made available for comment at the end of October.  We now release the final version.

We hope that this revision will be useful for people implementing ILS’s, ILS interaction layers, and discovery applications, and enable easier interoperation between ILS’s (existing and planned) and innovative discovery applications of all kinds.  We look forward to seeing implementations of these recommendations (some of which are already in progress), and further progress towards interoperability and improved discovery of the knowledge resources of libraries.

I’d like to re-echo my thanks I made on the release of  our “1.0 revision”  back in the summer, and thank everyone who helped write, comment on, and support this recommendation.

And now, I think I’ve got some implementation work to do…

October 30, 2008

DLF ILS Discovery Interfaces: Revised recommendation draft open for comments

Filed under: architecture,discovery,libraries — John Mark Ockerbloom @ 3:41 pm

Today we released a draft of “revision 1.1” of the ILS Discovery Interfaces recommendation. As I discussed in my previous post, this revision is intended to clarify the implementation of the Basic Discovery Interfaces recommended for integrated library systems (ILS’s), and make them more useful for discovery applications.

On the DLF ILS Discovery Interfaces web site, you’ll find the revision draft and the accompanying schema, along with the initial official recommendation (or “revision 1.0”). My last post included a summary of the major changes from version 1.0.

We’d like to give folks a chance to comment on the changes before we make them official. We’ll take comments until November 18, shortly after the end of the DLF Fall Forum, so folks wanting to go to our birds of a feather session on implementing the recommendations can talk with us there and still have some time to send in written comments. (Or, you can send them in ahead of time so we can think on them at the forum.) Comments may be emailed to me, and I will pass them along to the rest of the task group. There’s also still the open Google Group for discussions.

I’m hoping we’ll start to see Basic Discovery Interfaces implementations, clients, and test suites soon based on the new recommendations and schema. They’re not that different from version 1.0, but should be more useful. I’m working on revising my example implementation now, and hope to see more implementations in the not too distant future. And I look forward to hearing interested people’s thoughts and comments as well.

October 20, 2008

Update on ILS-Discovery Interface work

Filed under: architecture,discovery,libraries — John Mark Ockerbloom @ 4:01 pm

It’s been a while since I posted about the official release of the Digital Library Federation’s ILS Discovery interface recommendation. Marshall Breeding recently posted a useful update on the further development of the interfaces at Library Technology Guides. As the chair of the ILS-DI task group, which is now charged with some followup work described in Marshall’s article, I’d like to add some further updates.

As Marshall mentions, the DLF convened a meeting in August inviting potential developers of the ILS-Discovery interfaces to discuss implementations of recommendations of the DLF’s ILS-Discovery Interface task group. In the course of the discussion, a few changes were suggested and generally agreed upon by the participants. Updating the recommendation was not the main purpose of the meeting, but as we discussed things, it became clear that some clarifications and small updates to the recommendation would be helpful for producing more consistent and useful implementations of the Basic Discovery Interfaces, the interoperability “Level 1” that was agreed to in the Berkeley Accord.

The ILS-DI task group is therefore preparing a slight revision, to be known as “version 1.1” of the recommendation. A draft of this revision will be released for comment shortly, and will include the following changes, summarized here to give developers some idea of what to expect:

  • For the HarvestBibliographicRecords and HarvestExpandedRecords functions, it will be clarified that the function should return the records that are available for discovery. (That is, suppressed records and others that might be in the ILS but aren’t intended for discovery will not be shown, except possibly as deleted records as described below).
  • Support for the OAI-PMH binding for these functions will be noted as required. (That is, it must be supported for full ILS-BDI compliance; other bindings can be supported too.) It will also be noted that Dublin Core is a minimum requirement for returned records (as it is for OAI-PMH in general), and that if MARC records exist in the ILS (or are produced by it), MARC XML should also be available.
  • We also will require some level of support for deleted records (which includes records no longer available for discovery), to make it feasible for discovery apps to keep in sync with the ILS’s records via incremental harvesting. We’ll note that ILSs should document how long they keep deleted-record information.
  • For GetAvailability, the simple availability schema defined in the document will be noted as required. (That is, it should be returned for full ILS-BDI compliance; other schemas can be supported too if asked for and supported.) There was some talk at the August meeting about completely dropping the alternative NCIP and ILS-Holdings schemas as replies to GetAvailability, because of their complexity. The draft at this point doesn’t go that far, but it will specify the simple availability schema as the default, and the required, schema to support in the ILS-BDI profile.
  • That simple availability schema will also be augmented slightly to include an optional location element, distinct from the availability-message element. Location was the one specific data field that many implementors said was essential to include that wasn’t in the original schema.
  • We will also add a request parameter to GetAvailability for specifying whether bib or item-level availability is desired if a bib. identifier is given. (Formerly the server had the option of choosing the level in that case; there was a strong sentiment in discussions that the client to be able to specify this.)
  • We expect to leave GoToBibliographicRequestPage alone.

The new draft will be released shortly, and be open to public comment for at least a couple of weeks before we make a last edit for an official release. Feedback is welcome and encouraged, and public discussion can take place in the ILS-DI Google Group, among other places

The new draft will be accompanied by a revised XML schema. The current schema, reflecting the original or “version 1.0” official recommendation, can be found here. For the location of the new one (which is not yet posted), substitute “1.1” for “1.0” in the schema URL. (We intend to keep the old schema up for a good while after the new one is posted, for compatibility with implementations based on the original recommendation.)

I will also be leading a Birds of a Feather session at the upcoming Digital Library Federation fall forum in Providence next month. This will be an opportunity for developers of interfaces implementing the DLF’s ILS-Discovery interface recommendations to present their work to others, ask and answer questions about the recommendations and their implementations, and discuss further development initiatives and coordination. If you’d like us to set aside some time to show or discuss a particular initiative or project you’re working on, let me know.

Watch this space and the ILS-DI Google Group for further developments. And if you can come to the session at DLF in November, I hope we’ll have an interesting and enlightening discussion there as well.

(Update, Oct. 30: The draft of the revision is now out for comment.)

October 9, 2008

Surpassing all records

Filed under: architecture,discovery,preservation,repositories — John Mark Ockerbloom @ 10:47 pm

What will happen to all the White House emails after George W. Bush leaves office in January? Who will take charge of all the other electronic records of the government, after they’re no longer in everyday use? How can you archive 1 million journal articles a month from dozens of different publishers? Can the virtual world handle the Large Hadron Collider’s generation of 15 petabytes of data per year without being swallowed by a singularity? And how can we find what we need in all these bits, anyway?

These were some of the digital archiving challenges discussed this week at the Partnerships in Innovation II symposium in College Park, Maryland. Co-sponsored by the National Archives and Records Administration and the University of Maryland, the symposium brought together experts and practitioners in digital preservation for a day and a half of talks, panels, and demonstrations. It looked to me like over 200 people attended.

This conference was a sequel to an earlier symposium that was held in 2004. Many of the ideas and plans presented at the earlier forum have now grown into fruition. The symposium opened with an overview of NARA’s Electronic Records Archives (ERA), a long-awaited system for preserving massive amounts of records from all federal government agencies, that went live this summer. It’s still in pilot mode with a limited number of agencies, but will be importing lots of electronic records soon, including the Bush administration files after the next president is inaugurated.

The symposium also reviewed progress with older systems and concepts. The OAIS reference model, a framework for thinking about and planning long-term preservation repositories, influences not only NARA’s ERA, but many other initiatives and repositories, including familiar open source systems like Fedora and DSpace. Some of the developers of OAIS, including NASA’s Don Sawyer, reviewed their experiences with the model, and the upcoming revision of the standard. Fedora and DSpace themselves have been around long enough to be subjects of a “lessons learned” panel featuring speakers who have built ambitious institutional repositories around them.

The same panel also featured Evan Owens of Portico discussing the extensive testing and redesign they had to do to scale up their repository to handle the million articles per month mentioned at the top of this post. Heavily automated workflows were a big part of this scaling up, a strategy echoed by the ERA developers and a number of the other repository pracitioners, some of whom showed some interesting tools for automatically validating content, and for creating audit trails for certification and rollback of repository content.

Networks of interoperating repositories may allow digital preservation to scale up further still. That theme arose in a couple of the other panels, including the last one, dedicated to a new massive digital archiving initiative: the National Science Foundation‘s Datanet. NSF envisions large interoperating global networks of scientific data that could handle many Large Hadron Colliders worth of data, and would make the collection, sharing, reuse, and long-term preservation of scientific data an integral part of scientific research and education. The requirements and sizes of the grants are both prodigious– $20 million each to four or five multi-year projects that have to address a wide range of problems and disciplines– but NSF expects that the grants will go to wide-ranging partnerships. (This forum is one place interested parties can find partners.)

I gave a talk as part of the Tools and Technologies panel, where I stressed the importance of discovery as part of effective preservation and content, and discussed the design of architectures (and example tools and interfaces) that can promote discovery and use of repository content. My talk echoed in part a talk I gave earlier this year at a Palinet symposium, but focused on repository access rather than cataloging.

I’m told that all the presentations were captured on video, and hopefully those videos, and the slides from the presentations, will all be placed online by the conference organizers. In the meantime, my selected works site has a PDF of the slides and a draft of the script I used for my presentation. I scripted it to make sure I’d stay within the fairly short time slot while still speaking clearly. The talk as delivered was a bit different (and hopefully more polished) than this draft script, but I hope this file will let folks contemplate at leisure the various points I went through rather quickly.

I’d like to thank the folks at the National Archives and UMD (especially Ken Thibodeau, Robert Chadduck, and Joseph Jaja) for putting on such an interesting and well-run symposium, and giving me the opportunity to participate. I hope to see more forums bringing together large-scale digital preservation researchers and practitioners in the years to come.

June 20, 2008

Now it’s official

Filed under: architecture,libraries — John Mark Ockerbloom @ 12:17 pm

As I hoped, the good news was announced by Peter Brantley of the Digital Library Federation while I was away in Canada: the recommendations of the ILS-Discovery interface task group, which we’ve been talking about and drafting over the last many months online and off, have now been officially released. You can find the official release on the DLF website. We’ll be putting some supplementary information on there shortly as well; for now, you can still find background and supplementary material on our wiki.

I’d like to thank the members of the task group for all their work in putting the recommendation together; the Digital Library Federation for sponsoring this work; our steering group (Dale Flecker, Robert Wolven, Marty Kurth, Terry Ryan, and especially Peter Brantley) for all sorts of help and support in making this initiative viable; the Penn Libraries for supporting my chairing the task group this past year (as well as hosting one of the early meetings); the vendors that signed the Berkeley Accord for meeting with us and agreeing to support the basic discovery interface functions describes in our recommendation; and the many library folks, developers, and vendors that gave us suggestions and publicity.

We’ve intended the recommendations to be a first step in an ongoing process of supporting interoperability between the online data and services of libraries and a wide range of discovery applications. The recommendations we produced give fairly detailed proposals for a basic level of interoperability, and more open-ended proposals for higher levels. But you should only spend so long on proposals before it’s time to shift emphasis onto implementing them. With the official version now out, I hope we can start implementing these functions in earnest. (And once we’ve accumulated some experience with implementations, I hope that folks will revisit and refine the recommendations to further help things along.)

Locally, we already have one demonstration implementation, and we hope to now work on getting the basic functions implemented for our actual ILS.) And I hope that many others will be working on or using implementations soon. The DLF is now planning a developer’s workshop for folks interested in implementing the ILS-DI recommendations, which hopefully will convene later this summer. There should also be online forums of various kinds to support folks who are interested in implementing the recommendations or using them in their application. Exactly how these forums will develop over time remains to be seen; but for now, the ILS-DI Google Group is one good place to look for news and discussion of activities related to the ILS-DI recommendation.

I’m thankful myself for having the opportunity to work with so many good people on this project, and look forward to getting to work on implementations, and to continuing the conversations that have started to make the most of library resources and services.

June 5, 2008

A break, and coming attractions

Filed under: architecture,online books,reading — John Mark Ockerbloom @ 11:16 pm

I’m about to head off to the wilds (okay, the farms) of Saskatchewan to relax with family on a much-welcomed break. I’ve got to the point in packing where we’re trying to figure out which books to bring. (Which involves some careful selection to narrow it down to the number of books we can bring on the ever-more-limited-space airlines without excess baggage problems.)

I leave the ILS-Discovery Interface work in good hands, and there should be good news shortly (hopefully, quite shortly, and well before I return) for folks who are interested in this initiative. I’ll have more to say on what comes next after I get back. Also after I come back, a couple of weeks from now, I’ll be picking up on the repositories series I started last month, with a review of the what-why-who-and-where of the various kinds of repositories that libraries may find of use.

Online book fans may also be interested in following a debate going on now about ebook publishing, business models, and piracy. Author David Pogue had a Times Blog post a couple of weeks ago giving his reasons for not issuing electronic editions of his titles, that drew a long set of reader comments. Now Adam Engst has posted an interesting and detailed rebuttal, where he describes his own sales successes with his ebooks (piracy notwithstanding).

You might also enjoy “Reading sets you free”, an article posted about a month ago by K. G. Schneider (who I had the pleasure of meeting in person recently at a NISO discovery forum.) I was reminded of it again just now as I was trying to think of what books the kids might bring. As in the picture accompanying her article, both of them are very much read-under-the-covers kids at this point, as were both their parents. We’re all looking forward to spending a lot of time conversing with each other and with our books these next couple of weeks.

May 29, 2008

Views of possible future architectures of cataloging

Filed under: architecture,sharing — John Mark Ockerbloom @ 10:37 pm

PALINET convened a symposium today here in Philadelphia on the future of cataloging. There was a full turnout, with over 150 library professionals attending. It appeared that the organizers had to scramble a bit to distribute lunch to the large crowd. I waited for a few minutes in a line that hardly seemed to move at all, and then some logjam cleared, enabling us all to get our food in short order. (I did notice that by the time I picked up my own box lunch, no one was checking the tickets that specified what food we were entitled to take.) Would that all our cataloging projects could resolve their workflow and backlog issues so quickly.

The opening keynote was by Karen Calhoun, now at OCLC, whose controversial 2006 report for the Library of Congress touched off a fierce debate among librarians over what kinds of changes should take place in library catalogs. Her address at this symposium was less controversial, and dealt with transitions in the work of the folks that catalog and manage collections. My Penn colleague Beth Picknally Camden took part in the followup panel, remarking on the “perpetual beta” viewpoint that we’re encouraging in our library as we shift to new responsibilities and strategies. Also on the panel were Diane Hillmann (at Cornell until recently), and Christine Schwartz, whose blog, Cataloging Futures, is well worth following if you’re interested in future directions of library catalogs. (Besides the ongoing posts, its “key resources” column gives a useful overview of many of the current debates on cataloging.)

The symposium also provided an opportunity to learn more about the Library of Congress’s 2007 recommendation on the future of bibliographic control (in a presentation by Nancy Fallgren), as well as FRBR and RDA, two bibliographic standards proposed to become the new basis for bibliographic description (and featured in a presentation by John Attig.) I would have loved to go to both talks, preferably one right after another– if nothing else, the contrasting points of view would have been interesting. (The LC report recommended that work on RDA be suspended, in part due to concerns about the practicality of FRBR.) Alas, they were at the same time, so I attended Attig’s talk, which covered material less familiar to me than the contents of the LC report.

I also had to miss Christine di Bella’s talk on special collections cataloging to give my own talk. I’m not firmly settled into any established camp in the cataloging debate, but I’ve noticed that architectural issues– information architecture, systems architecture, and social architecture– underly many of the ongoing cataloging debates, and aren’t always explicitly considered or fleshed out. So I tried to address some of them in my talk, using projects I’m involved with such as subject maps, ILS discovery interfaces, and PennTags, as examples of designs that aim for a more robust catalog architecture. The slides I’ve used, which include pointers to more information about all these projects, are now posted on my Selected Works website. PALINET also intends to put the audio and slides of all of us who spoke on their website (though I’m not quite sure where they will end up, or whether they will be all accessible to the general public.)

I was happy to see several people raise the importance of freely sharing cataloging data, something that’s all too often hindered by existing contracts, and which severely impairs the community’s ability to improve the catalog collectively. Diane Hillmann was particularly eloquent on this issue, urging people to consider open source-like business models that support themselves by providing the best services, not by hoarding data. My talk also touched on “open data” issues. (And Karen Coyle recently blogged on an example of the kind of damage we’re inflicting on ourselves by not agreeing to share.) I did hear some encouraging hints suggesting that some aggregators might be moving towards more open sharing of commonly managed catalog records, as well as easier ways for the cataloging community to refine and improve on these records. We’ll see what happens.

This was my first PALINET symposium, and the first conference I’ve been to that focused specifically on cataloging issues. I’m very glad I went, and I thank PALINET for inviting me to speak (and running a smooth and enjoyable conference, lunch lines notwithstanding). If you’re interested in these issues, I hope you’ll find my talk slides of interest, and hope we’ll see more materials from the speakers online as well before long.

May 16, 2008

An implementation of the DLF’s Basic Discovery Interfaces recommendation

Filed under: architecture,online books — John Mark Ockerbloom @ 4:21 pm

The DLF’s ILS-Discovery interface recommendation work, which I’ve been leading, continues. We’re now in the process of producing the official recommendation, which I hope will be out soon. (Especially since I fully intend it to be out there before I head off to the great white North in early June.) And the May Library Gang podcast features a conversation with me and various other folks in libraries and the commercial world about the ILS-DI work and its implications.

You don’t have to wait until the official release, though, to start experimenting with the interfaces. I’ve now implemented the Level 1 recommendations for The Online Books Page, so folks can see what an implementation can look like to an application. (And you’re also free to just use the interfaces if you find the data and services useful, though I reserve the right to limit access to them if out server gets overloaded.) I’ve also put up a page with more information on the interfaces and how to use them.

I’m hoping we’ll see ILS-DI interfaces for standard ILSs as well before long (whether they’re provided by ILS vendors or library developers working on top of vendor interfaces.) We have some interest in having the interfaces on top of our Voyager catalog, though that would take a while longer to implement. The Online Books Pages implementation, though, shows how the interfaces aren’t just for ILS’s, but can also use data and services from other online digital collections.

If the recommended interfaces become sufficiently widely and uniformly supported, a discovery application could draw on a wide range of sources, both in a local library and beyond it, and let its users discover resources from any or all of them in a largely seamless fashion. Which I think is a great way to help readers take full advantage of the library resources we all make available for them.

In the meantime, I hope you find this example implementation useful. I’ll be happy to hear and answer questions and comments about it, and about the ILS-DI work in general.

April 24, 2008

ILS-Discovery interoperation: New recommendation draft, last call for comments

Filed under: architecture,libraries,Uncategorized — John Mark Ockerbloom @ 4:25 pm

The new draft of the ILS-Discovery recommendations I mentioned in my last post is now out. You can download it, and read more about it, on our task force wiki.

As I mentioned previously, we intend this draft to be the last release before the official final version. We don’t expect to change the basic recommended functions in major ways in the final draft, though there’s a lot more that can be said and done to promote interoperability beyond these first steps we’ve taken.

We are very interested in correcting and clarifying anything that is erroneous, ambiguous, or unclear, particularly in the Level 1 functionality we recommend. Comments can be emailed to me (“ockerblo” at “pobox.upenn.edu”) between now and Friday, May 9; I’ll pass them along to the task force working on this. We hope to do our final revisions and then release the official recommendation not long afterwards.

The task force will also be conducting a birds-of-a-feather discussion session at next week’s DLF Spring Forum in Minneapolis. The session will be held at 2:30 on Tuesday, April 29, in Greenway B on the second floor of the conference hotel. Topics of discussion include the Berkeley accord (the agreement with vendors and developers that informed this draft), the draft recommendation and its upcoming finalization, implementing the recommendation, and how to continue and build on efforts to promote and standardize interoperation between the ILS and discovery applications.

I’m still working on an example implementation of the Level 1 functions, but have been busy enough with the draft not to finish it yet (or blog about much else lately; there are some other topics in the pipeline, though!) I hope to point to that soon. And if you’re interested in our recommendation or what it’s trying to accomplish, I hope to hear from you. And maybe I’ll see you in Minneapolis next week.

« Previous PageNext Page »

The Rubric Theme. Blog at WordPress.com.


Get every new post delivered to your Inbox.

Join 102 other followers