Everybody's Libraries

July 31, 2010

Keeping subjects up to date with open data

Filed under: data,discovery,online books,open access,sharing,subjects — John Mark Ockerbloom @ 11:51 pm

In an earlier post, I discussed how I was using the open data from the Library of Congress’ Authorities and Vocabularies service to enhance subject browsing on The Online Books Page.  More recently, I’ve used the same data to make my subjects more consistent and up to date.  In this post, I’ll describe why I need to do this, and why doing it isn’t as hard as I feared that it might be.

The Library of Congress Subject Headings (LCSH) is a standard set of subject names, descriptions, and relationships, begun in 1898, and periodically updated ever since. The names of its subjects have shifted over time, particularly in recent years.  For instance, recently subject terms mentioning “Cookery”, a word more common in the 1800s than now, were changed to use the word “Cooking“, a term that today’s library patrons are much more likely to use.

It’s good for local library catalogs that use LCSH to keep in sync with the most up to date version, not only to better match modern usage, but also to keep catalog records consistent with each other.  Especially as libraries share their online books and associated catalog records, it’s particularly important that books on the same subject use the same, up-to-date terms.  No one wants to have to search under lots of different headings, especially obsolete ones, when they’re looking for books on a particular topic.

Libraries with large, long-standing catalogs often have a hard time staying current, however.  The catalog of the university library where I work, for instance, still has some books on airplanes filed under “Aeroplanes”, a term that recalls the long-gone days when open-cockpit daredevils dominated the air.  With new items arriving every day to be cataloged, though, keeping millions of legacy records up to date can be seen as more trouble than it’s worth.

But your catalog doesn’t have to be big or old to fall out of sync.  It happens faster than you might think.   The Online Books Page currently has just over 40,000 records in its catalog, about 1% of the size of my university’s.   I only started adding LC subject headings in 2006.  I tried to make sure I was adding valid subject headings, and made changes when I heard about major term renamings (such as “Cookery” to “Cooking”).  Still, I was startled to find out that only 4 years after I’d started, hundreds of subject headings I’d assigned were already out of date, or otherwise replaced by other standardized headings.  Fortunately, I was able to find this out, and bring the records up to date, in a matter of hours, thanks to automated analysis of the open data from the Library of Congress.  Furthermore, as I updated my records manually, I became confident I could automate most of the updates, making the job faster still.

Here’s how I did it.  After downloading a fresh set of LC subject headings records in RDF, I ran a script over the data that compiled an index of authorized headings (the proper ones to use), alternate headings (the obsolete or otherwise discouraged headings), and lists of which authorized headings were used for which alternate headings. The RDF file currently contains about 390,000 authorized subject headings, and about 330,000 alternate headings.

Then I extracted all the subjects from my catalog.  (I currently have about 38,000 unique subjects.)  Then I had a script check each subject see if it was listed as an authorized heading in the RDF file.  If not, I checked to see if it was an alternate heading.  If neither was the case, and the subject had subdivisions (e.g. “Airplanes — History”) I removed a subdivision from the end and repeated the checks until a term was found in either the authorized or alternate category, or I ran out of subdivisions.

This turned up 286 unique subjects that needed replacement– over 3/4 of 1% of my headings, in less than 4 years.  (My script originally identified even more, until I realized I had to ignore the simple geographic or personal names.  Those aren’t yet in LC’s RDF file, but a few of them show up as alternate headings for other subjects.)  These 286 headings (some of them the same except for subdivisions) represented 225 distinct substitutions.  The bad headings were used in hundreds of bibliographic records, the most popular full heading being used 27 times. The vast majority of the full headings, though, were used in only one record.

What was I to replace these headings with?  Some of the headings had multiple possibilities. “Royalty” was an alternate heading for 5 different authorized headings: “Royal houses”, “Kings and rulers”, “Queens”, “Princes” and “Princesses”.   But that was the exception rather than the rule.  All but 10 of my bad headings were alternates for only one authorized heading.  After “Royalty”, the remaining 9 alternate headings presented a choice between two authorized forms.

When there’s only 1 authorized heading to go to, it’s pretty simple to have a script do the substitution automatically.  As I verified while doing the substitutions manually, nearly all the time the automatable substitution made sense.  (There were a few that didn’t: for instance. when “Mind and body — Early works to 1850″ is replaced by “Mind and body — Early works to 1800“, works first published between 1800 and 1850 get misfiled.  But few substitutions were problematic like this– and those involving dates, like this one, can be flagged by a clever script.)

If I were doing the update over again, I’ll feel more comfortable letting a script automatically reassign, and not just identify, most of my obsolete headings.  I’d still want to manually inspect changes that affect more than one or two records, to make sure I wasn’t messing up lots of records in the same way; and I’d also want to manually handle cases where more than one term could be substituted.  The rest– the vast majority of the edits– could be done fully automatically.  The occasional erroneous reassignment of a single record would be more than made up by the repair of many more obsolete and erroneous old records.  (And if my script logs changes properly, I can roll back problematic ones later on if need be.)

Mind you, now that I’ve brought my headings up to date once, I expect that further updates will be quicker anyway.  The Library of Congress releases new LCSH RDF files about every 1-2 months.  There should be many fewer changes in most such incremental updates than there would be when doing years’ worth of updates all at once.

Looking at the evolution of the Library of Congress catalog over time, I suspect that they do a lot of this sort of automatic updating already.  But many other libraries don’t, or don’t do it thoroughly or systematically.  With frequent downloads of updated LCSH data, and good automated procedures, I suspect that many more could.  I have plans to analyze some significantly larger, older, and more diverse collections of records to find out whether my suspicions are justified, and hope to report on my results in a future post.  For now, I’d like to thank the Library of Congress once again for publishing the open data that makes these sorts of catalog investigations and improvements feasible.

May 6, 2010

Making discovery smarter with open data

Filed under: architecture,discovery,online books,open access,sharing,subjects — John Mark Ockerbloom @ 9:06 am

I’ve just made a significant data enhancement to subject browsing on The Online Books Page.  It improves the concept-oriented browsing of my catalog of online books via subject maps, where users explore a subject along multiple dimensions from a starting point of interest.

Say you’d like to read some books about logic, for instance.  You’d rather not have to go find and troll all the appropriate shelf sections within math, philosophy, psychology, computing, and wherever else logic books might be found in a physical library.  And you’d rather not have to think of all the different keywords used to identify different logic-related topics in a typical online catalog. In my subject map for logic, you can see lots of suggestions of books filed both under “Logic” itself, and under related concepts.  You can go straight to a book that looks interesting, select a related subject and explore that further, or select the “i” icon next to a particular book to find more books like it.

As I’ve noted previously, the relationships and explanations that enable this sort of exploration depend on a lot of data, which has to come from somewhere.  In previous versions of my catalog, most of it came from a somewhat incomplete and not-fully-up-to-date set of authority records in our local catalog at Penn.  But the Library of Congress (LC) has recently made authoritative subject cataloging data freely available on a new website.  There, you can query it through standard interfaces, or simply download it all for analysis.

I recently downloaded their full data set (38 MB of zipped RDF), processed it, and used it to build new subject maps for The Online Books Page.   The resulting maps are substantially richer than what I had before.  My collection is fairly small by the standards of mass digitization– just shy of 40,000 items– but still, the new data, after processing, yielded over 20,000 new subject relationships, and over 600 new notes and explanations, for the subjects represented in the collection.

That’s particularly impressive when you consider that, in some ways, the RDF data is cruder than what I used before.  The RDF schemas that LC uses omit many of the details and structural cues that are in the MARC subject authority records at the Library of Congress (and at Penn).  And LC’s RDF file is also missing many subjects that I use in my catalog; in particular, at present it omits many records for geographic, personal, and organizational names.

Even so, I lost few relationships that were in my prior maps, and I gained many more.  There were two reasons for this:  First of all, LC’s file includes a lot of data records (many times more than my previous data source), and they’re more recent as well.  Second, a variety of automated inference rules– lexical, structural, geographic, and bibliographic– let me create additional links between concepts with little or no explicit authority data.  So even though LC’s RDF file includes no record for Ontario, for instance, its subject map in my collection still covers a lot of ground.

A few important things make these subject maps possible, and will help them get better in the future:

  • A large, shared, open knowledge base: The Library of Congress Subject Headings have been built up by dedicated librarians at many institutions over more than a century.  As a shared, evolving resource, the data set supports unified searching and browsing over numerous collections, including mine.  The work of keeping it up to date, and in sync with the terms that patrons use to search, can potentially be spread out among many participants.  As an open resource, the data set can be put to a variety of uses that both increase the value of our libraries and encourage the further development of the knowledge base.
  • Making the most of automation: LC’s website and standards make it easy for me to download and process their data automatically. Once I’ve loaded their data, and my own records, I then invoke a set of automated rules to infer additional subject relationships.  None of the rules is especially complex; but put together, they do a lot to enhance the subject maps. Since the underlying data is open, anyone else is also free to develop new rules or analyses (or adapt mine, once I release them).  If a community of analyzers develops, we can learn from each other as we go.  And perhaps some of the relationships we infer through automation can be incorporated directly into later revisions of LC’s own subject data.
  • Judicious use of special-purpose data: It is sometimes useful to add to or change data obtained from external sources.  For example, I maintain a small supplementary data file on major geographic areas.  A single data record saying that Ontario is a region within Canada, and is abbreviated “Ont.”, generates much of my subject map for Ontario.  Soon, I should also be able to re-incorporate local subject records, as well as arbitrary additional overlays, to fill in conceptual gaps in LC’s file.  Since local customizations can take  a lot of effort to maintain, however, it’s best to try to incorporate local data into shared knowledge bases when feasible.  That way, others can benefit from, and add on to, your own work.

Recently, there’s been a fair bit of debate about whether to treat cataloging data as an open public good, or to keep it more restricted.  The Library of Congress’ catalog data has been publicly accessible online for years, though until recently only you could only get a little a time via manual searches, or pay a large sum to get a one-time data dump.  By creating APIs, using standard semantic XML formats, and providing free, unrestricted data downloads for their subject authority data, LC has made their data much easier for others to use in a variety of ways. It’s improved my online book catalog significantly, and can also improve many other catalogs and discovery applications.  Those of us who use this data, in turn, have incentives to work to improve and sustain it.

Making the LC Subject Headings ontology open data makes it both more useful and more viable as libraries evolve.  I thank the folks at the Library of Congress for their openness with their data, and I hope to do my part in improving and contributing to their work as well.

March 9, 2010

Implementing interoperability between library discovery tools and the ILS

Filed under: architecture,discovery — John Mark Ockerbloom @ 4:57 pm

Last June I gave a presentation in a NISO webinar about the work a number of colleagues and I did for the Digital Library Federation to recommend standard interfaces for Integrated Library Systems (the systems that keep track of our library’s acquisitions, catalog, and circulation) to support a wide variety of tools and applications for discovery.   Our “ILS-DI” recommendation was published in 2008, and encompassed a number of functions that some ILS’s supported.  But it also included many functions that were not generally, or uniformly, supported by ILS’s of the time.  That’s still the case today.

As I said in my presentation last June, “If we look at the ILS-DI process as a development spiral, we’ve moved from a specification stage  to an implementation stage.”  My hope has been that vendors and other library software implementers would implement the basics of what we recommended– as many agreed to– and the library community could progress from there.  This often takes longer to achieve than one might hope.

But I’m happy to report that the Code4lib community is now picking up the ball.  At this month’s Code4lib conference, a group met to discuss “collaboratively develop[ing] a middleware infrastructure” to link together ILS’s and discovery tools, based on the work done by the DLF’s ILS-DI group and by the developers of systems like Jangle and XC.  The middleware would help power discovery applications like Blacklight, VuFind, Summon, WorldCat Local, and whatever else the digital library community might invent.

I wasn’t at the Code4lib conference, but the group that met there to kick off the effort has impressive collective expertise and accomplishments.   It includes several members of the DLF’s ILS-DI group, as well as the lead implementors of several relevant systems.  Roy Tennant from OCLC Research is coordinating the initial activity, and Emily Lynema of the ILS-DI group has converted the Google groups space used by the ILS-DI group for the new effort.

And you’re welcome to join too, if you’d like to help out or learn more. “This is an open, collaborative effort” is how Roy put it in the announcement of the new initiative.  Due to some prior commitments, I’ll personally be watching more than actively participating, at least to begin with, but I’ll be watching with great interest.  To find out more, and to get involved, see the Google Group.

January 15, 2010

Concepts in catalogs: Where the data comes from

Filed under: architecture,discovery,formats,metadata,sharing,subjects — John Mark Ockerbloom @ 5:08 pm

I’ve now made a few posts about concept-oriented catalogs, describing the basic idea, showing some examples, and talking about the kinds of context they should provide for users.  As I mentioned in my first post, concepts in such catalogs are “first-class locuses of information to help readers find useful knowledge resources”.  The catalogs I’m describing include a variety of concepts (beyond the bibliographic record) that have data associated with them, and this data gives users a helpful context for finding appropriate knowledge resources.

As I said in my example post, “The concepts come from, and are maintained by, various groups of people…. [They]  may be derived in part from existing MARC bibliographic metadata (sometimes through automated analysis), but often draw from additional data sources.”

If you’ve worked in cataloging lately, you might be thinking, “that’s nice, but we’ve got our hands full just providing MARC catalog records for all the books and other stuff coming through the door now.   Where’s all this other ‘concept’ data going to come from?  And how will it be practical to use and maintain?”

In this post, I’d like to take a stab at answering those questions.  I’ll draw a lot from my experience with subject maps, but a lot of what I say should apply to other kinds of concept data as well.

The conceptual data behind subject maps consists of annotations on different subjects, and links between related subjects.  A lot of what I need to build these maps can simply be reused from existing data. In particular, the Library of Congress Subject Headings system (LCSH) provides a large set of subjects with standardized names.  We also have a set of authority records associated with those subjects that gave alternate names, notes, and links to related subjects.

To make it practical to build a subject map for this data, I bulk-loaded authority records from our local catalog.  While the Library of Congress Authorities are more up to date than our local catalog, I could only look up records there one at a time, through an interface designed for manual browsing.  Fortunately, since then the Library of Congress has provided ways to download subject authority data in bulk.  It’s in a format that omits some details, but it should still help fill out our maps when we start including these records as well.  Because our library and the Library of Congress are both using a common system of identifiers for subjects, as well as compatible formats for expressing subject relationships, I’ll be able to combine our authority information with theirs to provide useful maps.  The identifiers we use are not always in sync; LCSH subject terms do get renamed and discontinued from time to time.  But the cross-references in LCSH authority records, which often include the old terms as aliases of the new terms, help reduce the pain involved in moving from old terms to newer terms.

Subject maps built just on authority records turn out to be pretty generic, and not as useful as they could be.  To make them more useful, we need more data.  As I describe in more detail in this white paper, I also analyze our bibliographic corpus to see what subject terms we actually use in our catalog, look at the structure of those terms (which are often coordinated from multiple components), and also look at correlations between terms that get used together in the same bibliographic records.  This analysis lets me create additional useful relationships between subjects. In short, I use automated analysis of a large data corpus to create new concept data from existing data.

In order to link together the many subjects that have geographic aspects, I need some extra data that isn’t in authority records.   Once I created a data record that noted that “Pennsylvania” is a US state that gets abbreviated “Pa.” in some subject headings, I was able to build all kinds of relationships between “Philadelphia (Pa.)” and related subjects, none of which are directly stated in the authority records for these subjects, but all of which can be derived by automated analysis.  (It helps that subject terms in LCSH have a fairly well-defined structure that’s amenable to lexical analysis.)  A couple hundred other brief geographic data records are enough to let users zoom in and out of locations all over the globe.  So a small amount of well-designed and curated supplementary data can often enhance lots of concepts, with minimal maintenance cost.

While I can easily zoom in and out between the US, Pennsylvania, Philadelphia, and locations within Philadelphia, I’d need more data to move side to side.  I don’t have any data, for instance, that tells me that Philadelphia is right next to Camden, New Jersey.  But fortunately, I can mine external data sources to find this information.  I recently read about a source of public domain global map data, for instance, that I (or any other geographic-concept catalog builder) could use to link subjects or other resources to a world map.

Increasing amounts of public data are distributed online.  If the data is public domain, or available with a liberal license, I don’t have to worry about legal roadblocks to downloading it, analyzing it, and using it in my own work.  Sharing data helps everyone build not only smarter catalogs, but smarter applications of all kinds.

Data sharing does not always happen painlessly.  I may have different concepts, or different names for concepts, than someone else whose data I might find useful.  We may have different ideas about how to structure our data.  But there are now systems that provide links between different names, and crosswalks between different structures that can help bridge the gap between my data and that of others.

With large enough corpuses of data to draw on, I can even make use of unstructured information from large groups of ordinary users.  For example, LibraryThing’s tag cloud displays a number of terms that are useful to include in one’s own library catalog.  Not all of them are formally defined subjects, but they’re used enough that we should expect most of them to be used in patron searches.  It should be possible to analyze the cloud and the things tagged in the cloud to associate many informal terms with particular subjects or library resources.

To summarize, it becomes much easier to derive the data needed for concept-oriented catalogs if

  • We have stable (or at least smoothly evolving) identifiers for concepts
  • We can use, swipe, and reuse a large domain of [meta]data for concept analysis (including automated analysis)
  • We carefully consider what additional concept data would enhance our services, and use standard, recognized forms to represent it
  • We have correspondences and crosswalks between different concept identifiers and formats
  • We share our concept data (and bibliographic data in general) as openly and broadly as possible
  • And we share information, expertise, and code that supports the innovative, useful catalogs we build.

There’s a non-trivial technical infrastructure implied by these requirements.   But it’s one that we can build.  (Quite a bit of it’s in place already.)  A lot of it depends on a healthy social infrastructure to create, maintain, share, and work with all the data and services that we create and adopt.  I hope to talk more about this social infrastructure in future posts.

January 7, 2010

Content and context in concept-oriented catalogs

Filed under: architecture,discovery,metadata — John Mark Ockerbloom @ 4:45 pm

2010 has been designated the year of cataloging research by the Association for Library Collections and Technical Services.  So it’s a good time for me to continue my series on concept-oriented catalogs.  Before I talk about specific ways to implement them, I’d like to give some attention to what they look like, and the ways they interact with users.

If you’ve explored the links from my post giving examples of such catalogs, you may have noticed that there’s no single look and feel across the catalogs.  Most of them diverge substantially from the familiar “search box followed by uniform list of hits” paradigm we see with Google and other generic search engines.  But what should such catalogs provide, beyond the simple hit list?  And what do different well-designed concept-oriented catalogs have in common, from a user’s point of view?

Generally speaking, concept-oriented catalog displays are more complex than traditional catalogs.  Since they use a variety of concepts to help users find knowledge resources, the displays will generally show both a set of resources, and some information about other concepts in the catalog.

In theory, a catalog could start by just showing concepts, and requiring users to choose a particular concept before displaying the resources associated with it.  That’s in fact how many traditional library catalogs work, when users try to browse by author or by subject: you just see lists of headings, and you have to click through each heading individually before you can see what resources are associated with it.  But forcing this kind of interaction gets in people’s way unnecessarily.   I think that’s why I don’t see a lot of this sort of browsing  in library catalogs, and why the most frequently used Internet search engines take pains to show useful resource results as quickly as possible.

So good concept-oriented catalogs will feature resource “hits” quickly and prominently, just as other good catalogs will.  Concept-oriented catalogs will generally also show other concepts that provide context for these knowledge resources.   They can do this in a variety of ways.  They may mix in “concept” hits with “resource” hits (as with these search results for “Hava Nagila” in the Freedman archive, which combine hits for tracks, albums, and compositions).  They may display concept-related information alongside resource hit information.  (In some displays, like in WorldCat Identities’ display of Charles Dickens, resource hits are displayed after information about the concept. In other displays, like the Online Books Page subject map for “Prisoners of war”, they’re literally side by side.)  Or they may have complementary displays that overlay concepts and resources.  Consider, for instance, this Philadelphia neighborhood history display, where resources are represented as push-pins that overlay a background of geographic representations, a display also used in many Google Maps applications.

In designing a concept-oriented catalog, one needs to avoid confusing users with over-complex, under-explained displays.  Rather, the catalog should work with the user by providing an easily understood context for their search. Users should be given a clear idea of what concepts they’re looking at (e.g. by clearly naming and describing them), and what resources are associated with those concepts (e.g. by showing the resources along with the concepts, in some sensible organization).

In well-designed concept-oriented catalogs, the concepts and the resources complement each other.  The concepts, and the information displayed about them, helps users understand and choose appropriate resources.  And the resources, in turn, can help users understand what a concept is about, and what is known about the concept.  Looking over the available resources, the conceptual information, and mentions of related concepts, users can decide whether they want to delve more deeply into a particular set of concepts, or move to different concepts for further investigation.  (Facet-oriented displays let you do all this, for instance, and are useful even when they provide no information about concepts other than their names.)

In the Internet age, we also shouldn’t assume that users just stick to our own catalogs to do their research.  There’s a whole Web’s worth of context out there.  Concept-oriented catalogs should link out to (or include outright) external resources that can help users with concepts and resources, as appropriate.  Going the other way, they should also make it easy for external resources to stably link to a particular concept (or concept set) in the catalog.  (It’d be great if someone looking up an article on Wikipedia or another database could see a prominent link to “Stuff your library has on this topic”.)

I’ve mentioned or implied a number of design rules in what I’ve said above.  I’d like to point out 4 of them in particular, for a moment:

  • Use clear names for your concepts.
  • Make your concepts linkable via the Web.
  • Provide useful information about concepts when people come across them.
  • Include links to knowledge resources, other concepts, and external resources, so people can find more things.

Now I’d like you to look at the 4 rules for linked data by Tim Berners-Lee, inventor of the Web.

Notice a resemblance?  I didn’t start out trying to make my design requirements match Sir Tim’s principles, and I don’t necessarily think that Linked Data and its associated technologies are the be-all and end-all of information management.   But there are enough points in common to suggest some interesting synergies between the design of concept-oriented catalogs and the work of the Linked Data community.

And with that to think on, I’ll take my leave till next time, when I’ll talk more about the data that powers concept-oriented catalogs.

December 10, 2009

Some concepts and their catalogs

Filed under: architecture,discovery,libraries,metadata — John Mark Ockerbloom @ 6:28 pm

This is the second of a series of think-aloud posts on what I’m calling “concept-oriented catalogs”; catalogs that, as Lorcan Dempsey aptly describes them, go “beyond the bibliographic record”.  This post will present examples of concept-oriented catalogs, describe the concepts they use, and describe some of the features that make them work.

Reflections on the story so far

As I stated in my first post, a catalog helps a user get from some set of concepts (ideas, citations, words, people, places, etc.) of the information they’re seeking, to some useful knowledge resources (books, articles, web sites, songs, etc.)  A concept-oriented catalog uses a variety of concepts, not just those of knowledge resources themselves, as first-class entities to help locate useful resources.

Note that this is essentially a functional view of a catalog, from a user’s perspective, as a means of knowledge discovery.   Library catalogs also serve a number of other important functions.  In particular, they also manage the inventory of resources the library has acquired and makes available. That function is necessarily resource-oriented.  Some concept-oriented catalogs don’t need to be resource-oriented; Wikipedia, for example, maintains no inventory of the sites it links to from its concept-based articles.  A concept-oriented library catalog, however, will probably also need to be resource-oriented to fulfill all its functions.  That’s fine– the two qualities are not mutually exclusive.

Concepts can come from a variety of places; from libraries, from experts, and from ordinary readers.   I’ll show examples of all three.

Fun with FRBR

I noted in the last post that current library catalogs are largely based around the entities that FRBR describes as manifestations and items.  FRBR defines a number of other entities as well. These entities can also provide useful focuses for concept-oriented catalogs.

Fiction Finder, mentioned in my last post, is not the only example of a work-oriented catalog.  A recent status report from the OpenLibrary Project indicates that they are moving to make their catalog work-oriented as well.  The Amazon book catalog, and its aggregation of various editions of a book (and their reviews) on one catalog page, is oriented around expressions in the FRBR sense.

In practice, the line between works and expressions tends to be blurry in catalogs.  But we’re definitely seeing more catalogs present search results at these higher bibliographic levels. (See, for instance, the Worldcat.org search results for War and Peace, which include various “view all editions and formats” links.) Catalogs that can’t easily and sensibly aggregate their manifestations of works or expressions will be at a competitive disadvantage over catalogs that can.

Numerous catalogs also provide special information for authors, whether persons or corporate bodies.  I mentioned WorldCat Identities in my previous post; a commenter to that post noted the New Zealand Electronic Text Centre’s catalog, which also aggregates useful information on its authors, as well as other concepts.

Google Maps and other geospatial platforms provide the basis for many catalogs oriented around places, mediating access to information resources of all types, from local histories to violent incident reports, to hospital ratings. Catalog interfaces can also be built around events, as demonstrated in MIT’s Simile Timelines widget and in Google’s Living Stories news tracker.  Objects, concepts (which in FRBR-ese denote abstract subjects, rather than the broader use I’m making of “concept” in this post), and other kinds of subjects are the focus of the subject map-based catalogs I mentioned in my last post.

Catalogs need not limit their focus to one kind of concept. They often get more interesting when users can move between different kinds of concepts.  For instance, if you look up the author Fanny Jackson Coppin on The Online Books Page, you’ll see that there are not only resources by her, but resources about her.  If you look for the latter, you’ll also discover that Coppin was an African-American teacher in Philadelphia, and then be able to follow further links to more books about African American teachers, or Philadelphia, or other related subjects as you see fit.  Or, if you started a search with African American teachers in mind, you’ll find that concept linked to Coppin.  So users can move back and forth between particular subjects, and people that are relevant to them, as they browse the catalog.

What makes this sort of navigation work?  Part of it involves analysis of existing bibliographic records: in this case, catalogs like The Online Books Page (as well as others like WorldCat Identities) analyze patterns of subject headings to relate particular people to particular subjects.  Having common identifiers helps as well: the authority-controlled string “Coppin, Fanny Jackson” is used both for author and subject metadata, making it possible to link author- and subject-based concepts.  Subject maps also require additional data beyond just bibliographic records. Mine use a variety of data sources, including authority records and a small but essential set of records I created for certain geographical entities.

Consult the experts

Librarians have lots of understanding and experience working with FRBR-like concepts. But users also find value in many other concepts.  A recent research report from Project Information Literacy on information seeking patterns of college students included some interesting findings on favored starting points.  Wikipedia, as I expected, was a very popular starting point for everyday research, but for course-related research, the most popular starting point, edging out not only Wikipedia but even Google itself, was course readings.

Course readings are highly focused guideposts for academic research.  The list of readings is a set of knowledge resources chosen carefully by the instructor to educate students about the subject of the course.  The readings themselves typically have bibliographies, or at least lists of references, that support their own assertions and suggest further avenues of research.  Essentially, the syllabus and the readings represent a careful curation by subject experts of important knowledge resources for a particular topic.  The value of that curation often extends well beyond a particular class.  At Penn, many of the reading lists originally developed for a class get adapted into research guides on our library’s Web site, helping others researching in similar areas.  And interesting things also start to happen as you aggregate multiple scholars’ reading and citation lists, as we’ve seen with services like PennTags (originally designed, and still frequently used, for class projects), CiteULike, and Google Scholar.

There are also many collections assembled and curated by non-faculty experts.  The curation of many of these collections often involves a rich set of concepts appropriate to the focus of each collection.  Consider, for instance, the Freedman Jewish Sound Archive, curated by a lawyer and his wife and now housed at Penn.   The catalog for the archive is oriented around several important concepts: among them songs that are musical compositions, tracks that record performances of the songs,  albums that contain the tracks, sheet music in which songs are published, and artists that have various kinds of relationships with each of those other concepts.  Each of these concepts has an expressive metadata schema that includes relations to the other concepts, and to resources that can be read or listened to onsite or online.  The catalog is not only a guide to knowledge resources, but a valuable knowledge resource in its own right.

Because of the complexity and interrelationship between concepts, we had to write a specialized interface to bring out the full expressiveness of the catalog.  Simply providing a flat search index of tracks was not enough.   But many of the special concepts used to provide depth for this catalog can also be mapped to some extent into more common bibliographic data structures and interfaces.  And new technologies I’ll discuss later on can make it easier to build, and share data from, these sorts of specialized catalogs.

There are a lot more scholars, and expert curators, of all types, than there are professional librarians.  And they often know more about knowledge resources in their areas of interest than we do.  We can potentially build much more broad, conceptually rich, and carefully curated catalogs, if we develop effective ways to work with them.

Power to the people

Ordinary readers or scholars might not have the information science training or metadata expertise that professional librarians have.  But they can still help us greatly in finding useful knowledge resources, just through their ongoing reading and commentary, if we have some way of tracking and aggregating what they do.

Social software gives us a way to do that, and its benefits have been widely discussed in recent years.  Social software introduces the user as a first-class concept that can be associated with particular descriptions and knowledge resources.  Implicitly or explicitly, users build up collections of resources that they have noted, possibly using tags to describe them, or posts to comment on them.  The tags and posts are typically informal expressions, rather than controlled terms or highly structured records.  Tags can often describe resources in more accessible language, or to greater specificity, than established controlled descriptive terms.  And users often find useful resource recommendations from other users who share their interests.

Many social communities have built up around different user groups, types of resource, or styles of communication. PennTags (oriented around Penn scholars), Flickr (oriented around photos and other images), and Twitter (oriented around short real-time posts) are just a few examples.  I’m far from the only person who relies heavily on the users I follow on Twitter for reading tips and current awareness.   Longer-form posts that review and invite comments on books and other works can also be very helpful in finding useful knowledge resources.

Social software has been making its way into libraries for a few years now.   Many library catalogs, ours included, now let users tag and annotate bibliographic records.  The catalogs provide a smattering of new functions based on this tagging, but tagging is essentially an add-on, not fully integrated into the catalog.

Catalogs designed from the ground up to be socially powered can look quite different.  We’re starting to see some of them now.  One notable “social catalog” is LibraryThing, which has been developed by Tim Spalding, a staff of about a dozen professionals, and a membership of nearly a million readers.  Tim gave a talk to librarians in October called “What is Social Cataloging?“.   In the talk, he describes and demonstrates the features and the concepts of his catalog (including many of the concepts I’ve mentioned in this post), and the many benefits that emerge when hundreds of thousands of readers collaboratively catalog their personal libraries.  The video of the full talk runs just under an hour, and is well worth watching.

LibraryThing does not provide all of the functionality you’d find in a good research library catalog.  But it provides many useful forms of guidance that most traditional catalogs lack, and it also suggests some ways in which libraries and their users can join forces in building comprehensive, user-focused, concept-oriented catalogs.   (Much of the bibliographic metadata used in LibraryThing comes from traditional library catalog records; and LibraryThing in turn now sells a service that embeds some of the other information it aggregates back into traditional library catalog displays.)

Recap and coming attractions

In this post, I’ve shown how catalogs can use a wide variety of concepts to help users find resources.  The concepts come from, and are maintained by, various groups of people, including librarians, scholars, collectors and other domain experts, and lots and lots of everyday readers.  The catalog concepts may be derived in part from existing MARC bibliographic metadata (sometimes through automated analysis), but often draw from additional data sources.  The catalogs may require new user interface designs, going beyond the standard search-box and list-of-hits paradigm, for users to take full advantage of their concepts.

The examples I’ve shown should demonstrate the versatility of concept-oriented catalogs, and also suggest some of the challenges of implementing them.  In future posts, I hope to discuss the technologies, data models, system architectures, and social structures that can help make useful concept-oriented catalogs practical for libraries to build and maintain.

December 4, 2009

Understanding concept-oriented catalogs

Filed under: architecture,discovery,libraries,metadata — John Mark Ockerbloom @ 6:03 pm

I’ve recently been thinking a fair bit about the future of library catalogs, particularly after reading Diane Hillmann’s review of the recent Library of Congress-commissioned Study of the North American MARC Records Marketplace.  Hillmann laments that the study tries to propose economic tweaks to the current system of MARC bibliographic record distribution, but doesn’t consider the more basic questions of whether this sort of distribution is what cataloging should be focusing on in the first place, or how the nature of cataloging (broadly conceived) in the Internet age is already  changing.  She concludes “The change we need is not really about records, or catalogers; it’s a new way to think about information and added value.”

I agree with a lot of what she has to say in the review, but I also worry that, by itself, what she recommends might be too easily dismissed by library planners.   They might imagine this sort of thinking as utopian, airy speculation with little relevance to the work they have to do now, getting books and other resources into their catalogs more efficiently. But in fact, new ways of cataloging and adding value to knowledge are already being practiced in many places online, and can be practically built on much of what libraries are already doing.  So I’d like to spend a few blog posts thinking out loud about the potential I see developing for catalogs, and practical ways for realizing that potential.

What does a catalog do?

Let’s start with the basics: What are people doing when they’re using a library catalog, or engaging in other kinds of search?  Typically, they have in mind some concept [1], or combination of concepts, that they want to find information about.  Their concepts might not be fully formed or understood at the start of a search, and people may not be able to ideally express the concepts in ways their interface expects, but the concepts are there.

As they search, users often will refine their concepts, change the way  they express them, explore related concepts, or altogether shift the concepts they’re interested in.  They do this as they see what happens when they search, what resources come up, and what other relevant concepts (such as particular people, works, or focuses of study) become apparent.  But the user’s ultimate goal is to obtain useful knowledge.  A catalog, then, is a way of helping people get from concepts to useful knowledge resources.

There are many kinds of concepts people that searchers might be thinking of.  Familiar kinds of library concepts include books, people, places, and topics of study, but there are many additional important kinds that we’ll discuss later on.  Sometimes, users already have a concept of a specific knowledge resource.  They might have a citation they’re trying to follow, or a particular book they want to read.  Library catalogs tend to be particularly good at this kind of scenario, known in the trade as a “known item search“. [2]

This is not surprising, because today’s library catalogs tend to be strongly oriented around resources, and much less around other concepts.  They primarily use MARC bibliographic records (which describe particular manifestations, to use FRBR terminology), and holdings records (which describe particular items that can be accessed or borrowed).   Library science has meticulously defined and curated a number of other concepts: subjects with complex taxonomies; authors with wide-ranging authority control; uniform titles and series that link many bibliographic items together, and other bibliographic features that have carefully defined controlled vocabularies.

These concepts are represented in our MARC records, but as distinctly second-class entities.   They’re typically attributes of the records that are the focus of the catalog, rather than focused records in their own right.  The closest things to first-class non-resource concepts in most catalogs are authority records.  But those records often contain minimal information about the concepts they describe, they’re typically invisible to most catalog users, and most online catalog interfaces don’t effectively exploit the information those records do contain. [3]

Concepts as first-class entities in catalogs

One of the long-recognized strengths of research libraries is the knowledge and organization they have of their resources, and of the concepts that these resources represent.  Since a catalog maps concepts to resources, it should be possible for catalogs to represent important concepts more explicitly and expressively in the catalog, and use them as guideposts to help people find useful resources.  We’re starting to see some prototype catalogs of this sort develop in the library world: consider, for instance, OCLC’s Worldcat Identities, oriented around authors; or their Fiction Finder, oriented around works with multiple manifestations; or the Online Books Page’s subject map views (such as this one on alphabets).  In these catalogs, the concepts aren’t simply metadata attributes, or headings in a list of choices, but information-rich reference points for finding knowledge resources.  These systems are all what I would call concept-oriented catalogs: catalogs that use various concepts (and not just concepts of the resources themselves) as first-class locuses of information to help readers find useful knowledge resources.

The promise of concept-oriented catalogs is still largely unrealized in the library world.  Not only do current library catalogs often do little with conceptual knowledge, but proposals for future catalog architectures also often stick to keeping a tight focus on resources and make other concepts secondary.   For instance, Coyle and Hillmann note in their 2007 review of the proposed RDA cataloging standard that “the focus of RDA is called ‘the resource’ and the resource is a FRBR manifestation/item described using the same concept of a pre-coordinated ‘record’ as we find in AACR2″ (the older cataloging standard first published in 1978).  Similarly, the final project report for OLE, which aims to build a next-generation library architecture, treats resources as first-class entities, but not the descriptions of those resources.  The metadata describing knowledge resources are simply attributes of the resources, and not entities that can managed and shared in their own right. [4]

Concept-oriented catalogs at Internet scale

As more and more knowledge resources become available to users, via the expansion of the Internet, the streamlining of interlibrary loan services, and the mass digitization of print library materials, well-defined, well-documented, and well-connected concepts will become increasingly important for readers that want to find what is most useful to them in a sea of information.  While we will never have well-defined concepts for everything readers might be interested in, the concepts that have been defined by someone, somewhere, can serve as valuable guideposts for subsequent information seekers, if we’re smart about managing and using them.

This view might strike some longtime users of the Internet as hopelessly naive and ignorant of history.  After all, the Web started out in the 1990s with various conceptually organized catalogues like the WWW Virtual Library and the Yahoo Directory.  But most people soon forsook these in favor of search engines like Google that are much more comprehensive, and that work with arbitrary keywords instead of predefined concepts.   Why should we think that concept-oriented catalogs will work at the Internet scale if they’ve already been tried and rejected on the Web?

In fact, though, concept-oriented cataloging is still very widely used in online information seeking today.  It just isn’t the same kind of concept-oriented cataloging.  The most popular concept-oriented catalog online today doesn’t force readers to go through a particular concept hierarchy before they can get at the resources they want.  Instead, it typically shows up prominently whenever you do a Google search for one of the concepts it includes.  Each of its millions of concepts gets reviewed for naming, redundancy, and relevance, and can be easily linked to from elsewhere on the Web. Most of its concepts have links to various external online knowledge resources, as well as to related concepts.  The catalog also has ever-increasing amounts of harvestable, structured, semantic metadata about its concepts.  And it’s very often characterized, even by users who acknowledge its many flaws, as a useful starting point for finding information resources online.  Oh, and anyone can edit it.  You’ve probably figured out by now that I’m talking about Wikipedia.  While Wikipedia is not usually billed as a catalog, my description should make it clear that it does in fact serve that function, and in a concept-oriented way.

Coming attractions

I hope what I’ve written so far gives you a good idea of what concept-oriented catalogs are, and why they’re worth thinking about as we plan for the future of libraries.  In posts to come, I hope to discuss various examples of concept-oriented catalogs inside and outside the library world, and talk about how they work, what concepts (and related information) they focus on, and how they are built up and maintained. And I hope to show how we  construct useful, practical concept-oriented catalogs for the future, building both on the knowledge and expertise we have in libraries, and on the contributions of others.

(Update: The second post in the series is now available.)

Notes

[1] I’m using “concept” here in the broad sense of any kind of thing that might be the object of someone’s search, rather than the narrower sense used in FRBR and elsewhere of a particular kind of abstract subject.  It’s surprisingly difficult to find a straightforward word for this general idea that hasn’t already been claimed for some other purpose.

[2] Though even here, with the explosion of resources available online, we’ve found it useful to augment the traditional catalog with link resolvers to help people find resources like journal articles that we don’t specifically include in our own catalogs.

[3] I’ve discussed this at length in the past with respect to LCSH subjects; see my work on subject maps for some attempts to address this problem.  I’ll talk more about subject maps as one type of concept-oriented catalog in later posts.

[4] This may well change as OLE develops, and the data architecture is fleshed out.  I’m not directly involved in OLE at present, but I do work for one of the lead development partners.

June 10, 2009

Learn more about ILS discovery interfaces

Filed under: architecture,discovery,libraries — John Mark Ockerbloom @ 1:01 pm

I’m presenting today at a NISO webinar on interoperability, giving an overview of the work I did with a Digital Library Federation task group to produce recommendations for standard APIs for ILS’s supporting information discovery applications.

I’ll include a link to my presentation later today, after the webinar is over.   I’m also happy to answer questions here about the ILS-DI work.  (I’ve also covered that work here before in the blog.)

To help folks keep track of ILS-DI implementations and related activities, I’ve also created a new page on this site linking to the recommendation, implementations and followons, and related projects.  I’ve started it with just the basics, but plan to fill in more information shortly.

Update: I’ve now posted my slides and speaker notes.

January 28, 2009

Open catalog APIs and data: ALA presentation notes posted

Filed under: architecture,discovery,libraries,open access,sharing — John Mark Ockerbloom @ 3:48 pm

I’ve now posted my materials for the two panels I participated in at ALA Midwinter.

I have slides  available for “Opening the ILS for Discovery: The Digital Library Federation’s ILS-Discovery Interface Recommendations“, a presentation for LITA’s Next Generation Catalog interest group, where I gave an overview of the recommendations and their use.   At the same session, Beth Jefferson of BiblioCommons talked about some of the social and legal issues of sharing user content in library catalogs and other discovery applications.

And I have the slides and remarks I prepared for  “Open Records, Open Possibilities“, a presentation for the ALCTS panel on shared bibliographic records and the future of WorldCat.  In that one, I argue for more open access to bibliographic records, showing some of the benefits and sustainability strategies of open access models.

Karen Calhoun has also posted the slides from her presentation at that panel.  Peter Murray also presented; I haven’t yet found his slides online, but he’s blogging about what he said.  The fourth panelist, Brian Schottlaender, didn’t present slides, but instead gave thoughtful summaries and follow-on questions to some of the points the rest of us made.  From the audience, Norman Oder of Library Journal took notes and then wrote a useful report on the session.

I’d like to thank the organizers of these sessions, Sharon Shafer and Charles Wilt, for inviting me to speak, and my co-presenters for sharing their ideas and viewpoints.

January 15, 2009

Repository services, Part 2: Supporting deposit and access

Filed under: discovery,formats,repositories — John Mark Ockerbloom @ 6:01 pm

A couple of days ago, I talked about how we provided multiple repository services, and why an institutional scholarship repository needs to provide more than just a place to store stuff.  In this post, I’ll describe some of the useful basic deposit and access services for institutional scholarly repositories (IRs).

The enumeration of services in this series is based in part on discussions I’ve had with our scholarly communications librarian, Shawn Martin, but any wrong-headed or garbled statements you find here can be laid at my own feet.  (Whereupon I can pick them up, smooth them out, and find the right head for them.)

Ingestion:

One of the major challenges of running an institutional repository is filling it up with content: finding it, making sure it can go in, and making sure it goes in properly, in a manageable format, with informative metadata.  Among other things, this calls for:

  • Efficient, flexible, user-friendly deposit workflows. Most of your authors will not bother with anything that looks like it’s wasting their time.  And you shouldn’t waste your staff’s time either, or drive them mad, with needlessly tedious deposit procedures they have to do over and over and over and over again.
  • Conversion to  standard formats on ingestion. Word processing documents, and other formats tied to a particular software product, have a way of becoming opaque and unreadable a few years after the vendor has moved on to a new version, a new product, or that dot-com registry in the sky.  Our institutional repository, for instance, converts text documents to PDF on ingestion, which both helps preserve them and ensures wide readability.  (PDF is an openly specified format, readable by programs from many sources, available on virtually all kinds of computers.)
  • Journal workflows. Much of what our scholars publish is destined for scholarly journals, which in turn are typically reviewed and edited by those scholars.  Letting scholars review, compile, and publish those journals directly in the repository can save their time, and encourage rapid, open electronic access.   (And you don’t have to go back and try to get a copy for your repository when it’s already in the repository.)  Our BePress IR software has journal workflows and publication built into it.  Alternatively, specialized journal editing and publishing systems, such as Open Journal Systems, also serve as repositories for their journal content.
  • Support for automated submission protocols such as SWORD. Manual repository deposit can be tedious and error-prone, especially if there are multiple repositories that want your content (such as a funder-mandated repository, your own institution repository, and perhaps an independent subject repository.)  Manual deposit also often wastes people’s time re-entering information that’s already available online.  If you can work with an automated protocol that can automatically put content into a repository, though, things can get much better: you can support multiple simultanous deposits, ingestion procedures designed especially for your own environment that use the automated protocol for deposit, and automated bulk transfer of content from one repository to another.  SWORD is an automated repository deposit protocol that is starting to be supported by various repositories. (BePress does not yet support it, but we’re hoping they will soon).

From a practical standpoint, if you want a significant stream of content coming into your repository, you’ll probably need to have a content wrangler as well: someone who makes sure that authors’ content is going into the repository as intended. (In practice, they often end up doing the deposit themselves.)

Discovery:

You want it to be easy and enjoyable for readers to explore your site and find content of interest to them.  Here are a few important ways to enable discovery:

  • Search of full text and/or metadata, either over the repository as a whole, or over selected portions of the repository.  Full text search can be simple and turn up lots of useful content that might not be discovered through metadata search alone.  More precise, metadata-based searches can also be important for specialized needs.   Full text indexing is not always available (in some cases, you might only have page images), but it should be supported where possible.
  • Customization of discovery for different communities and collections.  Different communities may have different ways of organizing and finding things.  Some communities may want to organize primarily by topic, or author, or publication type, or date.  Some may have specialized metadata that should be available for general and targeted searching and browsing.  If you can customize how different collections can be explored, you can make them more usable to their audiences.
  • Aggregator feeds using RSS or Atom, so people can keep track of new items of interest in their favorite feed readers.  This needs to exist at multiple levels of granularity.   Many repositories give RSS feeds of everything added to the repository, but most people will be more interested in following what’s new from a particular department or author, or in a particular subject.
  • Search engine friendliness. Judging from our logs, most of the downloads of our repository papers occur not via our own searching and browsing interfaces, but via Google and other search engines that have crawled the repository.  So you need to make sure your repository is set up to make it easy and inviting for search engines to crawl.  Don’t hide things behind Flash or Javascript unless you don’t want them easily found.  Make sure your pages have informative titles, and the site doesn’t require excessive link-clicking to get to content.  You also need to make sure that your site can handle the traffic produced by search-engine indexers, some of which can be quite enthusiastic about frequently crawling content.
  • Metadata export via protocols like OAI-PMH.  This is useful in a number of ways:  It allows your content to be indexed by content aggregators; it lets you maintain and analyze your own repository’s inventory; and, in combination with automated deposit protocols like SWORD (and content aggregation languages like OAI-ORE), it may eventually make it much simpler to replicate and redeposit content in multiple repositories.

Access:

  • Persistent URIs for items. Content is easier to find and cite when it doesn’t move away from its original location.  You would think it would be well known that cool URLs don’t change, but I still find a surprisingly large number of documents put in content management systems where I know the only visible URIs will not survive the next upgrade of the system, let alone a migration to a new platform.  If possible, the persistent URI should be the only URI the user sees.  If not, the persistent URI should at least be highly visible, so that users link to it, and not the more transient URI that your repository software might use for its own purposes.
  • An adequate range of access control options for particular collections and items.  I’m all in favor of open access to content, but sometimes this is not possible or appropriate.  Some scholarship includes information that needs to be kept under wraps, or in limited release, temporarily or permanently.  We want to still be able to manage this content in the repository when appropriate.
  • Embargo management is an important part of  access control.   In some cases, users may want to keep their content limited-access for a set time period, so that they can get a patent, obey a publishing contract, or prepare for a coordinated announcement.  Currently, because of BePress’ limited embargo support, we sit on embargoed content and have to remember to put it into the repository, or manually turn on open access, when the embargo ends.  It’s much easier if depositors can just say “keep this limited access until this data, and then open it up,” and the repository service handles matters from there.

That may seem like a lot to think about, but we’re not done yet.  In the next part, I’ll talk about services for managing content in the IR, including promoting it, letting depositors know about its impact, and preserving it appropriately.

« Previous Page — Next Page »

Theme: Rubric. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 47 other followers