Everybody's Libraries

December 10, 2009

Some concepts and their catalogs

Filed under: architecture,discovery,libraries,metadata — John Mark Ockerbloom @ 6:28 pm

This is the second of a series of think-aloud posts on what I’m calling “concept-oriented catalogs”; catalogs that, as Lorcan Dempsey aptly describes them, go “beyond the bibliographic record”.  This post will present examples of concept-oriented catalogs, describe the concepts they use, and describe some of the features that make them work.

Reflections on the story so far

As I stated in my first post, a catalog helps a user get from some set of concepts (ideas, citations, words, people, places, etc.) of the information they’re seeking, to some useful knowledge resources (books, articles, web sites, songs, etc.)  A concept-oriented catalog uses a variety of concepts, not just those of knowledge resources themselves, as first-class entities to help locate useful resources.

Note that this is essentially a functional view of a catalog, from a user’s perspective, as a means of knowledge discovery.   Library catalogs also serve a number of other important functions.  In particular, they also manage the inventory of resources the library has acquired and makes available. That function is necessarily resource-oriented.  Some concept-oriented catalogs don’t need to be resource-oriented; Wikipedia, for example, maintains no inventory of the sites it links to from its concept-based articles.  A concept-oriented library catalog, however, will probably also need to be resource-oriented to fulfill all its functions.  That’s fine– the two qualities are not mutually exclusive.

Concepts can come from a variety of places; from libraries, from experts, and from ordinary readers.   I’ll show examples of all three.

Fun with FRBR

I noted in the last post that current library catalogs are largely based around the entities that FRBR describes as manifestations and items.  FRBR defines a number of other entities as well. These entities can also provide useful focuses for concept-oriented catalogs.

Fiction Finder, mentioned in my last post, is not the only example of a work-oriented catalog.  A recent status report from the OpenLibrary Project indicates that they are moving to make their catalog work-oriented as well.  The Amazon book catalog, and its aggregation of various editions of a book (and their reviews) on one catalog page, is oriented around expressions in the FRBR sense.

In practice, the line between works and expressions tends to be blurry in catalogs.  But we’re definitely seeing more catalogs present search results at these higher bibliographic levels. (See, for instance, the Worldcat.org search results for War and Peace, which include various “view all editions and formats” links.) Catalogs that can’t easily and sensibly aggregate their manifestations of works or expressions will be at a competitive disadvantage over catalogs that can.

Numerous catalogs also provide special information for authors, whether persons or corporate bodies.  I mentioned WorldCat Identities in my previous post; a commenter to that post noted the New Zealand Electronic Text Centre’s catalog, which also aggregates useful information on its authors, as well as other concepts.

Google Maps and other geospatial platforms provide the basis for many catalogs oriented around places, mediating access to information resources of all types, from local histories to violent incident reports, to hospital ratings. Catalog interfaces can also be built around events, as demonstrated in MIT’s Simile Timelines widget and in Google’s Living Stories news tracker.  Objects, concepts (which in FRBR-ese denote abstract subjects, rather than the broader use I’m making of “concept” in this post), and other kinds of subjects are the focus of the subject map-based catalogs I mentioned in my last post.

Catalogs need not limit their focus to one kind of concept. They often get more interesting when users can move between different kinds of concepts.  For instance, if you look up the author Fanny Jackson Coppin on The Online Books Page, you’ll see that there are not only resources by her, but resources about her.  If you look for the latter, you’ll also discover that Coppin was an African-American teacher in Philadelphia, and then be able to follow further links to more books about African American teachers, or Philadelphia, or other related subjects as you see fit.  Or, if you started a search with African American teachers in mind, you’ll find that concept linked to Coppin.  So users can move back and forth between particular subjects, and people that are relevant to them, as they browse the catalog.

What makes this sort of navigation work?  Part of it involves analysis of existing bibliographic records: in this case, catalogs like The Online Books Page (as well as others like WorldCat Identities) analyze patterns of subject headings to relate particular people to particular subjects.  Having common identifiers helps as well: the authority-controlled string “Coppin, Fanny Jackson” is used both for author and subject metadata, making it possible to link author- and subject-based concepts.  Subject maps also require additional data beyond just bibliographic records. Mine use a variety of data sources, including authority records and a small but essential set of records I created for certain geographical entities.

Consult the experts

Librarians have lots of understanding and experience working with FRBR-like concepts. But users also find value in many other concepts.  A recent research report from Project Information Literacy on information seeking patterns of college students included some interesting findings on favored starting points.  Wikipedia, as I expected, was a very popular starting point for everyday research, but for course-related research, the most popular starting point, edging out not only Wikipedia but even Google itself, was course readings.

Course readings are highly focused guideposts for academic research.  The list of readings is a set of knowledge resources chosen carefully by the instructor to educate students about the subject of the course.  The readings themselves typically have bibliographies, or at least lists of references, that support their own assertions and suggest further avenues of research.  Essentially, the syllabus and the readings represent a careful curation by subject experts of important knowledge resources for a particular topic.  The value of that curation often extends well beyond a particular class.  At Penn, many of the reading lists originally developed for a class get adapted into research guides on our library’s Web site, helping others researching in similar areas.  And interesting things also start to happen as you aggregate multiple scholars’ reading and citation lists, as we’ve seen with services like PennTags (originally designed, and still frequently used, for class projects), CiteULike, and Google Scholar.

There are also many collections assembled and curated by non-faculty experts.  The curation of many of these collections often involves a rich set of concepts appropriate to the focus of each collection.  Consider, for instance, the Freedman Jewish Sound Archive, curated by a lawyer and his wife and now housed at Penn.   The catalog for the archive is oriented around several important concepts: among them songs that are musical compositions, tracks that record performances of the songs,  albums that contain the tracks, sheet music in which songs are published, and artists that have various kinds of relationships with each of those other concepts.  Each of these concepts has an expressive metadata schema that includes relations to the other concepts, and to resources that can be read or listened to onsite or online.  The catalog is not only a guide to knowledge resources, but a valuable knowledge resource in its own right.

Because of the complexity and interrelationship between concepts, we had to write a specialized interface to bring out the full expressiveness of the catalog.  Simply providing a flat search index of tracks was not enough.   But many of the special concepts used to provide depth for this catalog can also be mapped to some extent into more common bibliographic data structures and interfaces.  And new technologies I’ll discuss later on can make it easier to build, and share data from, these sorts of specialized catalogs.

There are a lot more scholars, and expert curators, of all types, than there are professional librarians.  And they often know more about knowledge resources in their areas of interest than we do.  We can potentially build much more broad, conceptually rich, and carefully curated catalogs, if we develop effective ways to work with them.

Power to the people

Ordinary readers or scholars might not have the information science training or metadata expertise that professional librarians have.  But they can still help us greatly in finding useful knowledge resources, just through their ongoing reading and commentary, if we have some way of tracking and aggregating what they do.

Social software gives us a way to do that, and its benefits have been widely discussed in recent years.  Social software introduces the user as a first-class concept that can be associated with particular descriptions and knowledge resources.  Implicitly or explicitly, users build up collections of resources that they have noted, possibly using tags to describe them, or posts to comment on them.  The tags and posts are typically informal expressions, rather than controlled terms or highly structured records.  Tags can often describe resources in more accessible language, or to greater specificity, than established controlled descriptive terms.  And users often find useful resource recommendations from other users who share their interests.

Many social communities have built up around different user groups, types of resource, or styles of communication. PennTags (oriented around Penn scholars), Flickr (oriented around photos and other images), and Twitter (oriented around short real-time posts) are just a few examples.  I’m far from the only person who relies heavily on the users I follow on Twitter for reading tips and current awareness.   Longer-form posts that review and invite comments on books and other works can also be very helpful in finding useful knowledge resources.

Social software has been making its way into libraries for a few years now.   Many library catalogs, ours included, now let users tag and annotate bibliographic records.  The catalogs provide a smattering of new functions based on this tagging, but tagging is essentially an add-on, not fully integrated into the catalog.

Catalogs designed from the ground up to be socially powered can look quite different.  We’re starting to see some of them now.  One notable “social catalog” is LibraryThing, which has been developed by Tim Spalding, a staff of about a dozen professionals, and a membership of nearly a million readers.  Tim gave a talk to librarians in October called “What is Social Cataloging?“.   In the talk, he describes and demonstrates the features and the concepts of his catalog (including many of the concepts I’ve mentioned in this post), and the many benefits that emerge when hundreds of thousands of readers collaboratively catalog their personal libraries.  The video of the full talk runs just under an hour, and is well worth watching.

LibraryThing does not provide all of the functionality you’d find in a good research library catalog.  But it provides many useful forms of guidance that most traditional catalogs lack, and it also suggests some ways in which libraries and their users can join forces in building comprehensive, user-focused, concept-oriented catalogs.   (Much of the bibliographic metadata used in LibraryThing comes from traditional library catalog records; and LibraryThing in turn now sells a service that embeds some of the other information it aggregates back into traditional library catalog displays.)

Recap and coming attractions

In this post, I’ve shown how catalogs can use a wide variety of concepts to help users find resources.  The concepts come from, and are maintained by, various groups of people, including librarians, scholars, collectors and other domain experts, and lots and lots of everyday readers.  The catalog concepts may be derived in part from existing MARC bibliographic metadata (sometimes through automated analysis), but often draw from additional data sources.  The catalogs may require new user interface designs, going beyond the standard search-box and list-of-hits paradigm, for users to take full advantage of their concepts.

The examples I’ve shown should demonstrate the versatility of concept-oriented catalogs, and also suggest some of the challenges of implementing them.  In future posts, I hope to discuss the technologies, data models, system architectures, and social structures that can help make useful concept-oriented catalogs practical for libraries to build and maintain.

December 4, 2009

Understanding concept-oriented catalogs

Filed under: architecture,discovery,libraries,metadata — John Mark Ockerbloom @ 6:03 pm

I’ve recently been thinking a fair bit about the future of library catalogs, particularly after reading Diane Hillmann’s review of the recent Library of Congress-commissioned Study of the North American MARC Records Marketplace.  Hillmann laments that the study tries to propose economic tweaks to the current system of MARC bibliographic record distribution, but doesn’t consider the more basic questions of whether this sort of distribution is what cataloging should be focusing on in the first place, or how the nature of cataloging (broadly conceived) in the Internet age is already  changing.  She concludes “The change we need is not really about records, or catalogers; it’s a new way to think about information and added value.”

I agree with a lot of what she has to say in the review, but I also worry that, by itself, what she recommends might be too easily dismissed by library planners.   They might imagine this sort of thinking as utopian, airy speculation with little relevance to the work they have to do now, getting books and other resources into their catalogs more efficiently. But in fact, new ways of cataloging and adding value to knowledge are already being practiced in many places online, and can be practically built on much of what libraries are already doing.  So I’d like to spend a few blog posts thinking out loud about the potential I see developing for catalogs, and practical ways for realizing that potential.

What does a catalog do?

Let’s start with the basics: What are people doing when they’re using a library catalog, or engaging in other kinds of search?  Typically, they have in mind some concept [1], or combination of concepts, that they want to find information about.  Their concepts might not be fully formed or understood at the start of a search, and people may not be able to ideally express the concepts in ways their interface expects, but the concepts are there.

As they search, users often will refine their concepts, change the way  they express them, explore related concepts, or altogether shift the concepts they’re interested in.  They do this as they see what happens when they search, what resources come up, and what other relevant concepts (such as particular people, works, or focuses of study) become apparent.  But the user’s ultimate goal is to obtain useful knowledge.  A catalog, then, is a way of helping people get from concepts to useful knowledge resources.

There are many kinds of concepts people that searchers might be thinking of.  Familiar kinds of library concepts include books, people, places, and topics of study, but there are many additional important kinds that we’ll discuss later on.  Sometimes, users already have a concept of a specific knowledge resource.  They might have a citation they’re trying to follow, or a particular book they want to read.  Library catalogs tend to be particularly good at this kind of scenario, known in the trade as a “known item search“. [2]

This is not surprising, because today’s library catalogs tend to be strongly oriented around resources, and much less around other concepts.  They primarily use MARC bibliographic records (which describe particular manifestations, to use FRBR terminology), and holdings records (which describe particular items that can be accessed or borrowed).   Library science has meticulously defined and curated a number of other concepts: subjects with complex taxonomies; authors with wide-ranging authority control; uniform titles and series that link many bibliographic items together, and other bibliographic features that have carefully defined controlled vocabularies.

These concepts are represented in our MARC records, but as distinctly second-class entities.   They’re typically attributes of the records that are the focus of the catalog, rather than focused records in their own right.  The closest things to first-class non-resource concepts in most catalogs are authority records.  But those records often contain minimal information about the concepts they describe, they’re typically invisible to most catalog users, and most online catalog interfaces don’t effectively exploit the information those records do contain. [3]

Concepts as first-class entities in catalogs

One of the long-recognized strengths of research libraries is the knowledge and organization they have of their resources, and of the concepts that these resources represent.  Since a catalog maps concepts to resources, it should be possible for catalogs to represent important concepts more explicitly and expressively in the catalog, and use them as guideposts to help people find useful resources.  We’re starting to see some prototype catalogs of this sort develop in the library world: consider, for instance, OCLC’s Worldcat Identities, oriented around authors; or their Fiction Finder, oriented around works with multiple manifestations; or the Online Books Page’s subject map views (such as this one on alphabets).  In these catalogs, the concepts aren’t simply metadata attributes, or headings in a list of choices, but information-rich reference points for finding knowledge resources.  These systems are all what I would call concept-oriented catalogs: catalogs that use various concepts (and not just concepts of the resources themselves) as first-class locuses of information to help readers find useful knowledge resources.

The promise of concept-oriented catalogs is still largely unrealized in the library world.  Not only do current library catalogs often do little with conceptual knowledge, but proposals for future catalog architectures also often stick to keeping a tight focus on resources and make other concepts secondary.   For instance, Coyle and Hillmann note in their 2007 review of the proposed RDA cataloging standard that “the focus of RDA is called ‘the resource’ and the resource is a FRBR manifestation/item described using the same concept of a pre-coordinated ‘record’ as we find in AACR2″ (the older cataloging standard first published in 1978).  Similarly, the final project report for OLE, which aims to build a next-generation library architecture, treats resources as first-class entities, but not the descriptions of those resources.  The metadata describing knowledge resources are simply attributes of the resources, and not entities that can managed and shared in their own right. [4]

Concept-oriented catalogs at Internet scale

As more and more knowledge resources become available to users, via the expansion of the Internet, the streamlining of interlibrary loan services, and the mass digitization of print library materials, well-defined, well-documented, and well-connected concepts will become increasingly important for readers that want to find what is most useful to them in a sea of information.  While we will never have well-defined concepts for everything readers might be interested in, the concepts that have been defined by someone, somewhere, can serve as valuable guideposts for subsequent information seekers, if we’re smart about managing and using them.

This view might strike some longtime users of the Internet as hopelessly naive and ignorant of history.  After all, the Web started out in the 1990s with various conceptually organized catalogues like the WWW Virtual Library and the Yahoo Directory.  But most people soon forsook these in favor of search engines like Google that are much more comprehensive, and that work with arbitrary keywords instead of predefined concepts.   Why should we think that concept-oriented catalogs will work at the Internet scale if they’ve already been tried and rejected on the Web?

In fact, though, concept-oriented cataloging is still very widely used in online information seeking today.  It just isn’t the same kind of concept-oriented cataloging.  The most popular concept-oriented catalog online today doesn’t force readers to go through a particular concept hierarchy before they can get at the resources they want.  Instead, it typically shows up prominently whenever you do a Google search for one of the concepts it includes.  Each of its millions of concepts gets reviewed for naming, redundancy, and relevance, and can be easily linked to from elsewhere on the Web. Most of its concepts have links to various external online knowledge resources, as well as to related concepts.  The catalog also has ever-increasing amounts of harvestable, structured, semantic metadata about its concepts.  And it’s very often characterized, even by users who acknowledge its many flaws, as a useful starting point for finding information resources online.  Oh, and anyone can edit it.  You’ve probably figured out by now that I’m talking about Wikipedia.  While Wikipedia is not usually billed as a catalog, my description should make it clear that it does in fact serve that function, and in a concept-oriented way.

Coming attractions

I hope what I’ve written so far gives you a good idea of what concept-oriented catalogs are, and why they’re worth thinking about as we plan for the future of libraries.  In posts to come, I hope to discuss various examples of concept-oriented catalogs inside and outside the library world, and talk about how they work, what concepts (and related information) they focus on, and how they are built up and maintained. And I hope to show how we  construct useful, practical concept-oriented catalogs for the future, building both on the knowledge and expertise we have in libraries, and on the contributions of others.

(Update: The second post in the series is now available.)

Notes

[1] I’m using “concept” here in the broad sense of any kind of thing that might be the object of someone’s search, rather than the narrower sense used in FRBR and elsewhere of a particular kind of abstract subject.  It’s surprisingly difficult to find a straightforward word for this general idea that hasn’t already been claimed for some other purpose.

[2] Though even here, with the explosion of resources available online, we’ve found it useful to augment the traditional catalog with link resolvers to help people find resources like journal articles that we don’t specifically include in our own catalogs.

[3] I’ve discussed this at length in the past with respect to LCSH subjects; see my work on subject maps for some attempts to address this problem.  I’ll talk more about subject maps as one type of concept-oriented catalog in later posts.

[4] This may well change as OLE develops, and the data architecture is fleshed out.  I’m not directly involved in OLE at present, but I do work for one of the lead development partners.

December 1, 2009

Respecting failure: Some thoughts, and a proposal

Filed under: failure,libraries — John Mark Ockerbloom @ 8:02 pm

Last month’s Digital Library Federation forum involved a number of interesting discussions, both at the conference site and online.  This forum, different from previous ones, centered around discussions of strategies for innovation in libraries.  It also involved discussions of the future of DLF itself, which earlier this year ended its independent existence and was merged into CLIR.

One theme that quickly emerged in discussions was the importance of failure.  It’s a topic we often feel uncomfortable discussing, especially when we had a hand in whatever failed.  Part of the discomfort in the digital library community has to do with the dual nature of what many of us do: We manage programs and services, and we also try to innovate.  As managers, we don’t want our programs to fail.  (If that happens anyway, we’d at least like to avoid being blamed for the failure.)  And libraries have long-term ongoing service and preservation obligations that make certain kinds of catastrophic failure unacceptable.

But as innovators, we want to be open to failure as a way of learning.  “Fail faster!” is a common slogan of innovative labs and ventures, and knowing the “thousands of ways that don’t work” (part of a quote often attributed to Thomas Edison) help us better understand the ways that do.  I’ve mentioned before that my most widely-cited paper was written about the failure of a software development project I helped work on.  And a new scientific theory isn’t usually worth considering until it is capable of failure– that is, it makes definite predictions that subsequent observations can either confirm or refute.

If we are really serious about innovating, we need to respect failure, and leave room for it.  We need to let people try things that might not work, allow time for encountering dead ends, have contingency plans that let us continue to carry out our missions even as failures occur, and note both what worked and what didn’t in the things we try.  It’s especially useful to note things that we found didn’t work before they were obvious to others, since we might well save others a lot of time avoiding the same pitfalls.

How should these failures be reported, though?  It’s often easier and more gratifying to publish stories of success than stories of failure.  And if we talk about our failures in public, whether in a journal, at a conference of like-minded professionals, or even in a blog or tweet, we may have good reason to worry that it might hurt our own positions in our organizations, or the work that we do.

The question was raised at the DLF Forum whether future forums could be a “safe” place to discuss failures.  I’m not sure any public gathering can be an entirely safe place for that.  People you see are identifiable, and word tends to spread over time, particularly when a large number of people are present.

One alternative that’s been proposed to address this problem, and still make useful information about failure available to a wide audience, is to anonymize reports.  We can still learn a lot from failures in our field even if we don’t know exactly who failed and where.  And perhaps anonymity can produce more useful information about failure, by producing more “safe” places to talk about it.

So, I’d like to try a little experiment.  From now until the end of February, I invite folks involved in library-related failures to send me reports of their failures, for possible publication in this blog.  I will choose which ones to publish (and may or may not request revisions), but whether or not they get published, I will do my best not to disclose your identity.  (I can’t absolutely prevent email hacks or subpoenas, but I consider them unlikely.)  Please clearly note the start and end of the report you’re submitting for publication; I’ll assume any information that might help identify you or your organization in the middle of the report is information that you’re comfortable disclosing.  Reports should be sent to the email address shown on this page.

I’m interested in particular in failures of initiatives you were personally involved with, and what can be learned from the failures. While others may also be involved in the failures, I’m not particularly interested in reports intended primarily to focus blame on a particular third party.  You should use similar care to obscure the identities of others involved as you do in obscuring your own identity.

Some folks may feel more comfortable working with someone else, or under different selection criteria than the ones I use.  So I invite others to make similar offers if they see fit (and I might mention them here if I hear about any similar offers).

Will this produce something useful to the library community?   I don’t know yet, but it seems worth risking a bit of failure to find out.   If you have any suggestions about the proposal, or have some ideas of kinds of reports you’d like to see, feel free to mention them in the comments.  And if you have a report you’d like to make, send me email.

October 26, 2009

Promoting access to the best literature of the past

Filed under: online books,open access — John Mark Ockerbloom @ 3:24 pm

Last week saw widespread observance of Open Access Week 2009 .  The week primarily focused on opening access to current research and scholarship (though there’s also been a growing community working on opening access to teaching and learning content).  You can find lots of open access resources at the Open Access Directory.

Current scholarship is not spontaneously generated from the brain or lab of the writer.  Useful scholarship must understand and interpret past work, to be effective in the present.  In many fields, and not just the classical humanities, the relevant past work may stretch back hundreds or even thousands of years.  Current scholarship and study will be more effective if its source material is also made openly accessible, and if proper attention is drawn to the most useful sources.  And now is an especially opportune time for scholars of all sorts, professional and amateur, to get involved in the process.

This may seem a strange thing to say at a time when the digitization of old books and other historic materials is increasingly dominated by large-scale projects like Google and the Internet Archive.  With mass digitizers putting millions of public domain book and journal volumes online, and with a near-term possibility of millions more copyrighted volumes going online as well, how much of a role is left for individual scholars and readers?

A very important role, as it turns out.  Mass digitization projects can quickly produce large scale aggregations of pass content, but as many have pointed out, aggregation is not the same as curation, and as aggregations grow larger, being able to find the right items in a growing collection becomes increasingly important.  That’s what curation helps us do, and the large-scale digitizers are not doing a very effective job of it themselves.  Google’s PageRank algorithm may take advantage of implicit curation of web pages (through the choices of authors’ page links), but Google and other aggregators have had a much harder time drawing attention to the most useful books, scholarly articles, or other works created without built-in hyperlinks.

Sometimes this is because they haven’t digitized them, even as they’ve digitized inferior substitutes.  Over three years after Paul Duguid lamented the republication of a bowdlerized translation of Knut Hamsun’s Pan by Project Gutenberg, that version remains the only freely available one of this book available there, or at Google Books, or anywhere else online that I’ve found.   Even though an unexpurgated version of this translation was published before the bowdlerized version, no digitizer that I know of has gotten around to finding and digitizing it; and countless readers may have used the existing online copies without even knowing that they’ve been censored.  Extra bibliographic and copyright research may be necessary to determine whether a better resource is available for digitization, as it in this case.

Sometimes the content is digitized, but can’t be found easily.  Geoff Nunberg’s post on Google Books’ “metadata train wreck” shows plenty of examples of how difficult it can be to find and properly identify a particular edition in Google Books, much less figure out which edition is the best one to use.  I’ve commented in the past about the challenges of finding multi-volume works in that corpus.  And Peter Jacso has pointed out Google’s problems indexing current scholarship.  If you can’t find the paper or book you need for your research, your work will be no better than it would be if the source had never existed.

This is where scholars can potentially play a useful role.  We don’t individually digitize books by the thousands, but we do individually find, cite, and recommend useful sources, down to the particular edition, as we find them and use them in our own writings and teaching.  These citations and recommendations now often go online, in various locations.  It would be very useful to have these recommendations made more visible, and tied to freely available online copies of the sources cited, whenever legally possible. Sometimes, we also create or digitize our own editions of past works, with useful annotations, for our classes or our own work.  It would be very useful to have these made visible and persistent as well, whenever appropriate.

I hope that large resource aggregations will make it easier for scholars and others to curate the collections to make them more useful to their readers.  In the meantime, we can start with resources we have.  For example, on The Online Books Page, my catalog entry for Hamsun’s Pan notes its limitations.  My public requests page includes information on a better edition that could be digitized, by someone who has access to the edition and has some time to spare.  And my suggestion form is ready to accept links to better editions of this book, or to other online books that merit special attention.  Indeed, most of the books that I now add to my catalog derive from submissions made by various readers on this form, and I invite scholars to suggest the freely accessible books and serials that they find most useful for my catalog.

As the Little Professor notes in a recent post, the sort of bibliographic work I’ve described can be time-consuming but vitally important for making effective use of old sources, and that work has often not been done by anyone for many books outside the usual classical canons.  Yet it’s the sort of thing that scholars do, bit by bit, as part of their everyday work.  The aggregate effect of their curation and digitization, appropriately harnessed in open-access form, could greatly improve our ability to build upon the work of the past.

October 5, 2009

Remember this

Filed under: people,preservation,sharing — John Mark Ockerbloom @ 12:40 am

I am eating a sandwich at the end of Pier 14 in San Francisco.  The sun has set behind the downtown skyscrapers, and the colors in the sky are slowly fading to grey.  I’m not the only diner out here.  Pelicans soar close off the pier, about 100 feet above the water, and one by one dive straight down with a loud splash, resurfacing in a moment, ruffling their feathers and jerking their beaks to get down the fish they’ve caught.  Other splashes in the water come from seals surfacing for air.  As an orange-tinted full moon comes up over the East Bay hills and under the span of the Bay Bridge, I see a pair of seals surface side by side, with their mouths meeting as they float at the water’s surface for a few seconds.  I am delighted to see all this, so different from what I usually see at home, and at the same time I wish I could be back there with the people I love instead of alone here.

I don’t have a camera right now, or anything to draw with, so I can only record this scene in words and in memory.   When there was still sun shining low on Yerba Buena Island and the coastline to the east, there were several people out here with tripods and light umbrellas, photographing human couples standing against the pier railings, in each other’s arms.  Judging from the clothing and the poses, I suspect these shots are for wedding or engagement albums.  And I can understand the motivation.   When Mary and I were married, 14 years ago this month, we too had pictures taken of us against a striking background, in our case the bright orange and yellow trees of a Pennsylvania fall.  I see one of those pictures every time I return home. Remember this, the picture says, and it brings back memories of the vows we made to each other that day.  The words we said, and the way we looked when we said them, were not recorded in fixed form, but, God willing, will stay in our hearts as long as we live.

There are more memories recorded out on the pier.  Plaques along the rails quote lines of poetry by Lawrence Ferlinghetti and Thomas Lovell Beddoes about the bay I’m looking out on.  Ceramic tile art depicts boats that have plied its waters, from the early days of European exploration to the present.  A display on the sidewalk in front relates the history of the pier, the ferries that ran (and still run, in smaller numbers) from the terminal nearby, the freeway that was built and then removed again from the water’s edge, and some of the people who played a part in all of these developments.  Remember this, they say, and I bring bits back with me to record in words.

It’s a basic need that we have, as intelligent, reflective, and social creatures, to remember the things we’ve experienced, seen, and learned about.  We make records of these things in various forms, to help us remember, and to prompt others to remember as well.  They help us go beyond and above what’s immediately in front of us, telling us things we need to know, people we can relate to, pasts that were different, futures that can be better.

Technology can make it easier for us to record these things– and sometimes easier to lose them.  We took many pictures of our kids on digital cameras as they grew up, and kept hundreds of them on my laptop, which let me easily recall them and show them to friends and family when I traveled.  Then one day I was robbed of my laptop, without my having backed up my photo collection, and most of those pictures were lost.   I’ve also seen  many other personal and family memoirs posted on the Web, stay for a few years, and then vanish with the demise of the web site they were on.    I kept paper tapes of early BASIC programs I wrote in middle school for years after I had access to any device that could read them.  They’re gone now; I presume they were thrown out when my parents cleaned house sometime after I left home.

I know better now how to keep what remains.  Apple’s Time Machine makes it easy for me to incrementally back up my laptop every time I come home from work and plug a cheap external drive into my USB port.  The pictures of my kids that survived the laptop theft were mostly the ones that I had shared with others (either by copying them onto prints, or by putting them up on the Web). And the older family pictures that are most meaningful to us are ones where we know what the pictures represent, either because we are in them, or because others have told us, in person or in writing, who is in the pictures and the context in which they were taken.

I am here in San Francisco for Ipres 2009, a conference promoting the preservation of digital content.  There are a lot of smart, dedicated people scheduled to speak, and I hope to learn about new technologies and methods to help us preserve the content we want our libraries and their users to remember.

While some of these techniques may be complex, many of them are essentially elaborations on basic principles I’ve touched on in what I’ve related above: Help people record what’s important to them.  Make it easy for them to preserve these records in their everyday activity.  Encourage them to copy and share what they record, and allow others to build on them.  Make what they record easy to interpret, through informative description and straightforward formats.  And finally, try to understand and appreciate the connection between the record and the people for whom the record is important.

Which is why I sit now with my laptop in my hotel room, looking out on a bay that is now as dark as the night sky overhead, and trying to connect my experiences with the preservation challenges and proposals to come. Remember this, I mean to say.  It’s important.

September 17, 2009

Google Book settlement: Alternatives and alterations

Filed under: copyright,online books,open access — John Mark Ockerbloom @ 1:35 pm

In my previous post, I worried that the Google Books settlement might fall apart in the face of opposition from influential parties like the Copyright Office, and that such a collapse might deprive the public of meaningful access to millions of out of print books.

Not everyone sees it that way.  I’ve seen various suggestions of alternatives to the settlement for making these books available.  In this post, I’ll describe some of the suggested alternatives, explain why they don’t seem to me as likely to succeed on their own, and discuss how some of them could still go forward under a settlement.

Compulsory licenses

Both the Open Book Alliance’s court filings and the Copyright Office’s testimony mention the possibility of compulsory licensing, which essentially lets people use a copyrighted work without getting permission, provided that they meet standard conditions determined by the government.  Compulsory licenses already exist in certain areas, such as musical performances and broadcasts.  If I want to cover a Beatles song on my new record, I can, as long as meet some basic conditions, including paying a standard royalty.  The (remaining) Beatles can’t hold out for a higher rate, or say that no one else is allowed to cover the recordings they’ve released.

The Google Books settlement has some similarities to a compulsory license, but with some important differences, including:

  1. Book rightsholders can choose to deny public uses of their work, or hold out for higher compensation, which they generally can’t do under a compulsory license regime. (They have to explicitly request this, though.  So it’s really what one might call a “default” license.)
  2. The license has been negotiated through a court settlement rather than Congressional action. (This was one of the main complaints of the Copyright Office.)
  3. The license given in the settlement is granted only to Google, not to other digitizers. (This has justifiably raised monopoly concerns.)

I do have a problem with the last difference as it stands.  I’d like to see the license widened so that anyone, not just Google, could digitize and make available out of print books under the same terms as Google. But there are various ways we can get to that point from the settlement.  The Book Rights Registry created by the settlement could extend Google-like rights to anyone else under the same terms, as the settlement permits them to do.  The Justice Department could require them to do so as part of an antitrust supervision.  Or Congress could decide to codify the license to apply generally.  (They’ve done this sort of thing before with fair use and the first sale doctrine, both of which originated in the courts.)

If the settlement falls apart, though, negotiation over an appropriate license has to start over from scratch, and has to persuade Congress to loosen copyrights for benefits they might not clearly see. As I suggested in my previous post, Congress’ recent tendencies have heavily favored tightening, rather than loosening, copyright control.   And I haven’t yet seen a strong coalition pushing for laws granting compulsory (or default) licenses that are as broad as would be needed.

For instance, the Open Books Alliance’s amicus brief suggests the possibility of a compulsory license, but only as “but one approach”, and that suggestion seems as much aimed at getting hold of Google’s scans as licensing the book copyrights themselves.  Their front page at present shows no explicit advocacy of compulsory copyright licenses.  Perhaps they will unite behind a workable Google Books-style compulsory license proposal in the future, but I’m not counting on that.  (Update: Just after I posted this, I saw this statement of principles go up on the OBA site.  We’ll see what develops from that.)

The Copyright Office’s congressional brief also mentions but tries to damp down the idea.  It repeatedly characterizes compulsory licensing as something that Congress only does “reluctantly” and “in the face of marketplace failure”. But despite its strong words on other subjects, it does not appear concerned over whether we in fact have a marketplace failure around broad access to out-of-print books.

Orphan works legislation

The Copyright Office filing also suggests passing orphan works legislation (as have various other parties, including Google).  An orphan works limitation on copyrights would be nice, but it’s not going to enable the sort of large, comprehensive historical corpus that the Google Books settlement would allow.

As Denise Troll Covey has pointed out, the orphan works certification requirements recommended in last year’s bill, like many other case-by-case copyright clearance procedures, are labor-intensive and slow, and may be legally risky.  (In particular, the overhead for copyright clearance, not including license payment, can be several times the cost of digitization.)  Hence, these methods are not likely to scale well.  And they would not cover the many out-of-print books that aren’t, strictly speaking, orphans.  I don’t consider it likely that a near-comprehensive library  of millions of out-of-print 20th century books will come about by this route alone any time soon.

Even so, despite its limited reach, last year’s orphan works legislation was stopped in Congress after some creator organizations objected to it.  Some of the objectors, including the  National Writers Union and the American Society of Journalists and Authors, are now members of  the Open Book Alliance, which makes me wonder how effectively that group would act as a united coalition for copyright reform.

Private negotiation

Some critics suggest that Google and other digitizers simply negotiate with each rightsholder, or a mediator designated by each  rightsholder.   It’s possible that this actually might work for many future books, if authors and publishers set up comprehensive clearinghouses (like ASCAP and Harry Fox mediate music licensing).  If new books get registered with agents like these going forward, with simple, streamlined digital rights clearing, private arrangement could work well for future books both in-print and out-of-print.  Indeed, Google’s default settlement license privileges don’t apply to new books from 2009 onward.

But it’s much less likely that this will be a practical solution to build a comprehensive collection of past out of print books from the 20th and early 21st century, because of the sheer difficulty and cost of determining and locating all the current rightsholders of books long out of print.   The friction involved in such negotiation (involving high average cost for low average compensation) is too great.  Without the settlement and/or legal reform, we risk having what James Boyle called a “20th century black hole” for books.

Copyright law reform

As James Boyle points out, it would solve a lot of the problems that keep old books  in obscurity if books didn’t get exceedingly long copyrights purely by default.  It would also help if fair use and public domain determination weren’t as risky as they are now. I’d love to see all that come to pass, but no one I know that’s knowledgeable on copyright issues is holding their breath waiting for it to happen any time soon.

Moving forward

As I’ve previously mentioned, the settlement is imperfect.  It may well need antitrust supervision, and future elaboration and extension.  (And I’ve suggested some ways that libraries and others can work to improve on it.)  It’s still the most promising starting point I’ve seen for making comprehensive, widely usable, historic digital book collections possible.  I hope that we get the chance to build on it, instead of throwing away the opportunity.  In any case, I’d be happy to hear people’s thoughts and comments about the best way to move forward.

September 15, 2009

Google Books, and missing the opportunities you don’t see

Filed under: copyright,online books,open access — John Mark Ockerbloom @ 9:12 pm

The Google Books settlement fairness hearing is still a few weeks away, but in the last few weeks the deal has been talked and shouted about with ever-higher volume.  Still, it wasn’t until the other day, in a House Judiciary Committee hearing where US Copyright Register Marybeth Peters came loaded for bear, that I started thinking there was a significant likelihood that the settlement might fall apart.

There are a number of people in different communities, including libraries, who hope this  happens.   I’m not one of them.  I’m not a lawyer, so I can’t comment with authority on whether the settlement is sound law.  But I’m quite confident that it advances good policy.  In particular, it’s one of the best feasible opportunities to bring a near-comprehensive view of the knowledge and culture of the 20th and early 21st centuries into widespread use.  And I worry that, should the settlement break down, we will not have another opportunity like it any time soon.  The settlement has flaws, like the Google Books Project itself has, but at the same time, like Google Books itself, the deal the settlement offers is incredibly useful to readers, while also giving writers new opportunities to revive, and be paid for, their out of print work.

The potential

Under the status quo, millions of books are greatly under-utilized.  It isn’t just that people don’t have easy access to them; it’s that people don’t know that particular books useful to them exist in the first place.  I work in a library that has collected millions of volumes, many of which are hardly ever checked out. Not only would Google’s expanded books offerings give our users access to millions more books, but it would also make millions of books that we already own easier for our users to find and use effectively.

Want to know what books make mention of a particular event, ancestor, or idea?  With existing libraries, and good search skills, you might be able to find books, if any, that are written primarily about those things. But you’ll probably miss much other information on those same topics, information in works that are primarily about something else.  With expanded search, and the ability to preview old book content, it could be much easier to get a more comprehensive view on a topic, and find out which books are worth obtaining for learning more.

And if that’s a big advance for people in big universities like ours, it’s an even bigger step forward for people who have not had easy access to big research libraries.  Once a search turns up a book of interest, Google Books would offer a searcher various ways of getting that book: buying online access; reading it at their library’s computer (either via a paid subscription, or via a free public access terminal); buying a print copy; or getting an inter-library loan.  These options all involve various trade-offs of cost and convenience, as is the case with libraries today.  While one could wish for better tradeoff terms, the ones proposed still represent big advances from what one can easily do today.

And as with other large online collections like Wikipedia or WorldCat, or the Web as a whole, the advantages to large book corpuses like Google’s aren’t just in the individual items, but in what can be done with the aggregation.  I don’t know exactly what new kinds of things people will find to do with a near-comprehensive collection of  20th century books, but having seen all that people have done with other information aggregated on the Internet, I’m confident that there would be many great uses found, large and small.

The peril

If the Google settlement does fall apart, are we likely to see any collection like the one it envisions any time soon?  I’m not at all confident we will.  The basic problem is that, without some sort of blanket license, it’s impractical (and in the case of true orphan works, currently impossible) to clear all the copyrights that would be required to build such a collection.  This represents a failure in copyright law.  Instead of “promot[ing] the progress of science and useful arts”, as the Constitution requires, current US copyright law effectively keeps millions of out-of-print books in obscurity, not producing significant benefits either to their creators or to their potential users.

The current proposed Google Books settlement is, among other things, an attempt to get around this failure.  If the settlement fails, would the parties make a new agreement that would allow a readable collection of millions of post-1922 online books?  The divergence in the complaints I’ve seen (for instance, on one hand that the collection would cost readers too much, and on another hand that it would pay writers too little) suggest the difficulty of coming to a new consensus that satisfies all the parties, if negotiations have to start again from scratch.  And, if the arguments of the Copyright Office and some of the other parties carry the day, even if such an agreement were reached, the agreement could not be ratified by a court anyway.  Instead, it would require acts of Congress, and maybe even re-negotiations of international treaties.

Based on past history, there are two things that would make the government likely to reform copyright law to permit mass reuse of out-of-print books.  Ether there needs to be a clear example of the benefits of such a reform, or there needs to be a strong coalition pushing for such a reform.  Clear examples have usually come from businesses that are actually in operation; for example, the player piano roll industry that successfully persuaded Congress to streamline music copyright clearance in the previous century (or the Betamax that persuaded a slender majority of the Supreme Court to declare the VCR legal).

If the proposed Google Books library service goes online, even under a flawed initial settlement, it too could provide a compelling example to encourage general copyright reform.  But without such an example, it can be hard to move Congress to act.   It’s easy to undervalue the opportunities you don’t clearly see.

What about a strong coalition pushing for a reform in the law that would let anyone create the comprehensive online collections of out of print books I’d described?  I’d like to see one, but I haven’t yet.  (Yes, there’s the Open Book Alliance, but its members don’t seem to be distinctly allied in anything particular other than objecting to the settlement.)  In my next post, I’ll discuss reforms that might do the job, and the reasons I believe they would be difficult to enact without the settlement.

August 31, 2009

Why should reuse be hard?

Filed under: architecture,open access,publishing — John Mark Ockerbloom @ 11:04 pm

By far the most widely cited paper with my name on it is a 1995 paper on architectural mismatch.  The journal version of the paper was subtitled “Why reuse is so hard”.  It was a paper about failure, rather than success, which most researchers prefer to write about when they’re talking about their own work.  We discussed the problems we’d encountered trying to build a new software system from existing parts, and analyzed some of the reasons for the failures, and how systems could be improved in the future to make reuse easier.

The paper was unexpectedly well received, and was recently named as one of the most influential papers to appear in IEEE Software.  (I can’t claim too much credit for this myself; my adviser David Garlan and my fellow grad student Robert Allen rightly appear ahead of me in the author credits.)  ISI Web of Knowledge, which tracks the journal version of the paper, reports it’s been cited over 100 times in other journal articles; Google Scholar, which tracks both the journal version and the conference version that was published earlier the same year, reports hundreds more citations.

Google Scholar also reports an unexpected statistic: even though the journal version of a computer science paper is generally considered more authoritative than the earlier conference version (and rightly so, in our case), the conference paper has been cited even more often than the journal version.  Why is this?  I can’t say for sure, but there’s one important difference between the two versions:  the conference paper has been freely accessible on the web for years, and the journal paper hasn’t.  It’s in a highly visible journal, mind you– pretty much anywhere with a CS department subscribes to IEEE Software, and many individual computer practitioners subscribe as well.  So I suspect that most of the authors who cited our paper could have cited the journal paper (especially since it came out only a few months after the conference paper did).  But the conference paper was that much more easily accessible, and it was the one that got the wider reuse.

We’ve recently published a followup to our paper, appearing in the July/August issue of IEEE Software.  As we note in the followup, the problem of architectural mismatch has not gone away, but several developments have made it easier to avoid.   One of them is the great proliferation of open source software that has occurred since the mid-1990s, which provide a wide selection of software components to choose from in many areas, and “a body of experience and examples that clarify which architectural assumptions and application domains go with a particular  collection of software” (to quote from our paper).

Just as the growth of open source has made software easier to reuse, the growth of open access to research can make ideas and research results easier to reuse.  We saw that with our initial paper, I think, and I hope we’ll see it again with the followup. I’ve made it available as open access, with IEEE’s blessing.  Interested folks can check it out here.

July 14, 2009

For those wanting more drama in their lives

Filed under: copyright — John Mark Ockerbloom @ 9:15 pm

Thanks to the scanning services of Penn’s Schoenberg Center for Electronic Text and Image (SCETI), and a loan of over a dozen volumes from Stanford University Libraries, we have now posted online records of copyright renewals for drama (and works intended for oral delivery) up to 1968.  They can be found in page image form at our Catalog of Copyright Entries information page.

We’ve had copyright renewals for normal books, and periodicals, online for some years now.   But we’ve been missing renewal records for one important type of literary text: the kind primarily meant for performing or proclaiming, rather than reading.  And it turns out that copyright renewal records for works that aren’t books are harder to find than you’d expect: I could find no copies at all in the Philadelphia area for the years 1955-1968. So I’m grateful to Stanford for the loan of their volumes.  There are still a few more years of drama renewals left to digitize, but I’m hoping to find the later volumes in local libraries.

So far, we’ve only posted page images.  But I’m hoping that they’ll help people to make transcriptions and structured data records, as has been done with earlier page images we’ve posted.  I’ve tried to post easily readable but not-too-big copies of the pages online, to support manual searching and OCR.  Folks who need to see the masters, which have higher resolution but are substantially larger, can contact me.

Because Google is also starting to post original copyright registrations, it’s also possible to correlate renewals with original registrations, and look for interesting statistical phenomena and trends.  Already, for instance, one can see both that most copyrights were not renewed, and that the renewal rate can vary a lot by genre. For instance, in 1931, there were 552 “Class C” copyright registrations, covering lectures, sermons, addresses, and other works prepared for oral delivery.  (See page 1659 of this volume.)   There were also 5,993 drama registrations (“Class D”; see page 406 of this volume).  In 1959, around the time these copyrights were up for renewal, there were 780 Class D renewals, a renewal rate of about 13%.  There was only 1 Class C renewal in 1959, a renewal rate of well under 1%.  (A few previous years have slightly higher Class C renewal rates, but not by much.)

After I get through with drama, there’s one more class of renewal I’d like to see online as soon as possible: image renewals.  Only a few years worth of image renewals (from the early 1950s) are online, but what’s there suggests that the renewal rate for most published images was also quite low.  Since a large number of books and periodicals include images of one kind or another, knowing which images are still under copyright will be very important for folks who want to look at complete facsimiles of older works.

The proposed Google Books settlement, for instance, proposes blanking out anything in a book that might be an image (unless rights for that image have been cleared).  This could make many books substantially less useful than they would be if the images were cleared (especially since Google’s processing algorithms have to guess where images are, and may blank out not only images, but also text that can’t be easily auto-recognized.)

In the meantime, the drama renewals just posted will also help me look into some requests I’ve received for mid-20th-century plays and speeches whose status I’ve been unable to determine previously.  I hope it will prove useful for others as well.  Thanks again to SCETI and Stanford for these.

June 10, 2009

Learn more about ILS discovery interfaces

Filed under: architecture,discovery,libraries — John Mark Ockerbloom @ 1:01 pm

I’m presenting today at a NISO webinar on interoperability, giving an overview of the work I did with a Digital Library Federation task group to produce recommendations for standard APIs for ILS’s supporting information discovery applications.

I’ll include a link to my presentation later today, after the webinar is over.   I’m also happy to answer questions here about the ILS-DI work.  (I’ve also covered that work here before in the blog.)

To help folks keep track of ILS-DI implementations and related activities, I’ve also created a new page on this site linking to the recommendation, implementations and followons, and related projects.  I’ve started it with just the basics, but plan to fill in more information shortly.

Update: I’ve now posted my slides and speaker notes.

« Previous Page — Next Page »

Theme: Rubric. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.