metadata – Everybody's Libraries

Forward to Libraries update, and some thoughts on sustainability and scale

It’s been a while since I posted about Forward to Libraries, but if you’ve been following my Github repo, you may have noticed that it’s had a steady stream of updates and growth. If making connections across library collections or between libraries and Wikipedia interests you, the service is more comprehensive and wide-ranging than ever. Here’s an update:

Number of libraries

We now support forwarding to over 1,000 library systems worldwide, running dozens of off-the-shelf and custom-developed catalog and discovery systems. I’ve expanded coverage in all 50 states, Canada, the UK, and Australia, and have also added links to more countries in all inhabited continents. (Because the system currently focuses on Library of Congress headings, it works best with Anglo-American catalogs, but it yields acceptable results in some other catalogs as well, like the big Norway research library union catalog I just added today.) While major research libraries and big-city public libraries are well-represented, I’ve also been trying to add HBCUs, community colleges, rural library networks, and other sometimes-overlooked communities. And I respond quickly to requests from users to add their libraries.

Wikipedia coverage

Forward to Libraries can be used to make links between library collections, and to and from my millions-strong Online Books Page listings. But it’s also known for its interoperation with Wikipedia, where it knows how to link between over half a million distinct Library of Congress name and subject headings and their corresponding English language Wikipedia articles. The majority of these mappings are name mappings provided by OCLC’s VIAF, which in its most recent data dump includes over 485,000 VIAF identifiers that include both a Library of Congress Name Authorities identifier and an English Wikipedia article link. An additional 50,000 or so topical, demographic, and geographical LC subject headings are also mapped. These mappings derive from exact matches, algorithmic matching for certain kinds of heading similarities, and a manually curated mappings file that has grown over time to include more than 22,000 correspondences.

What you can do with this data

If you’re browsing a topic or an author’s works on the Online Books Page, you can follow links to searches for the same topic or author in any of the 1000+ libraries currently in my dataset. (If your library’s not one of them, just ask me to add it.) If the topic or the author is covered in one of the 500,000 corresponding Wikipedia articles the system knows of, you’ll also be offered a link to the relevant article.

If you’re browsing certain Wikipedia articles, you’ll also find links from them back to library searches in your favorite of those 1000+ library systems (or any other of those systems you wish to search). Right now those links use templates that must be manually placed, so there are only about 2500 Wikipedia articles with those links, but any Wikipedia editor can add the templates to additional articles. (A bot could potentially add more quickly, but that would require some negotiation with the Wikipedia community that I haven’t undertaken to date.) If you’re involved in a Wikipedia-library collaboration project (like this one), you may want to add one of these templates when editing articles on topics that are likely to have relevant source materials in multiple libraries. (The most common template used is the Library Resources Box, generally added to the External Links or Further Reading section of an article.)

If you’re interested in offering a similar library or Wikipedia linking service from your own catalog or discovery system, I’d be interested in hearing from you. You can either point to my forwarding service (using a standard linking syntax) or implement your own forwarder based on my code and data on Github. Right now it requires some effort and expertise to implement either method, but I’m happy to work with interested libraries or developers to make forwarding easier to implement.

Scaling and sustaining the service

The Forward to Libraries service still runs largely as a 1-person part-time project. (The Wikipedia templates are largely placed by others, and the service fundamentally depends on regularly updated data sets from OCLC, the Library of Congress, and Wikipedia, but I maintain the code and coordinate the central referral database myself.)

Part-time, 1-person projects raise some common questions: “Will they be sustained?” and “Can they scale?” I wasn’t sure myself of the answers to those questions when I started developing this service. Fortunately, I went ahead and introduced it anyway, and three-going-on-four years later, I’m happy to say that the basic service *is* in fact more sustainable and scalable than I’d thought it might be. The code is fairly simple, and doesn’t require a lot of updating to keep running. The main scale issues for the basic service have to do with the number of library systems and the number of topic mappings in the system, and those are both manageable.

The number of libraries turns out to be the more challenging factor to manage. Libraries change their catalogs and discovery systems on a regular basis, and when they do, search links that worked for their old catalogs often don’t work for the new ones. I have an automated tool that I run occasionally to flag libraries that return an error code to my search links; it’s not very sophisticated, but it does alert me to many libraries whose profiles I need to update, without my having to check all of them manually. (If you find any libraries where forwarding is no longer working, you can also use the suggestion form to alert me to the problem.) The more pressing scaling problem at the moment is the user experience: right now, when you’re asked to choose a library, the program shows a list of links to all 1000+ libraries currently in the system. That can be a bit much to handle, especially for users whose data is metered. Updating the library choice form to only show local libraries after the user selects the state, province or country they’re in will cut down on the data sent to the user; that may cost the user an extra click, but the tradeoff seems worth it at this point.

The number of topic mappings, on the other hand, has been easier to manage than I’d thought. VIAF publishes updated data files for names about once a month, and I can run a script over it to automatically update my service’s name heading mappings when a new file comes out. Likewise, Wikipedia is now providing twice-a-month dump files of its English language encyclopedia. I can download one of Wikipedia’s files in a couple of hours, and then run a script that flags any topical articles I map to that have gone away, changed their title, or redirected to other articles. I can then change my mappings file accordingly in under an hour. Library of Congress subject headings change as well, but they don’t change very fast. New and changed topical headings are published online about once a month, and one can generally add or change mappings from one of their updates within a few hours. I spend a few spare-time hours each month adding mappings for older subject headings, so if the current rate of LCSH growth holds, in theory I could cover *all* LCSH topical headings in my manual mappings file after some number of years. In practice, I don’t have to do this, especially as topical mappings also start to get collaboratively managed in places like Wikidata. (I’m not currently working with their data set, but hope to do so in the future.)

Broadening the service

While I can maintain and gradually grow the basic service with my current resources, broadening it would require more. Not every library, particularly outside the US, uses Library of Congress Subject Headings, so it would be nice to offer mappings to more subject ontologies and languages. Similarly, not everyone likes to work with Wikipedia (often with good reason), and it’d be nice to support links to and from alternative knowledge bases as well. The basic architecture of Forward to Libraries is capable of handling multiple library and web-based ontologies, but additional coding and data maintenance would be required. There are also various ways to build on the service to engage more deeply with libraries and current topics of interest; I’ve explored some ideas along these lines, but haven’t had the time to implement them.

Things continuing as they are, though, I should be able to maintain and grow the basic Forward to Libraries service for quite some time to come. I’m thankful to the people and the data providers that have made this service possible. And if you’re interested in doing more with it, or helping develop it in new directions, I’d be very glad to talk with you.

The value of catalogs in the linked data era: Two recent talks

I’ve recently uploaded two talks I gave this year to my Selected Works site. I presented “How Not to Waste Catalogers’ Time: Making the most of subject headings” at the Code4lib 16 conference in Philadelphia in March. “How to Read 100 Million Publications: VIVO and Comprehensive Open Publication Databases” was a talk I gave at VIVO 2016 last month. The versions I’ve deposited in Selected Works are PDF files that include both slides and notes, so you can see both what I showed, and (approximately) what I said during the presentations.

The talks can be seen as two complementary takes on the question “What’s the value of the catalog, and cataloging, in today’s networked environment?” In “How to Read 100 Million Publications” I highlight the value of an open, comprehensive database of the scholarly record that goes down to the article level. Such a database would be quite large (on the order of 100 million records, as the title suggests), and considerably finer-grain that present-day library-run catalogs, which typically catalog at the title level, and only add a little bit of volume information at the holdings level. But as I describe in the talk, 100-million-record-scale databases are now routinely maintained, and regularly replicated, online, and even a “just the facts” database of who published what articles where could be very helpful in promoting preservation, open access, corpus analysis, research networking systems, and a variety of other applications. It would not need to be built from scratch– the data required for such a database already largely exists online, though all too often in fragmentary or proprietary collections. But a number of these could be brought together and opened up, without requiring much additional cataloging labor. Shared-effort collaborations, like those that support databases like GOKb and SHARE, as well as more crowd-sourced projects like FictionMags or Wikipedia, could support the scale needed.

While “How to Read 100 Million Publications” proposes something new we could do with library catalogs, “How Not to Waste Catalogers’ Time” urges developers to pay close attention to what catalogers have already been doing with them– particularly the often-overlooked semantic richness of their subject cataloging. At the VIVO conference I saw a demo of a BIBFRAME application where a book’s subjects were represented by an ordered set of FAST properties. That’s easy enough to model as RDF linked data, but it omits the subject ordering that can improve relevance ranking, and lets readers distinguish the relative importance of subjects in a particular work. It also doesn’t accommodate the detailed on-the-fly subject categories that coordinated subdivisions allow. It’s possible to accommodate both of these in linked data, but it requires more complicated data models, and most catalogs don’t take full advantage of their semantic strengths. My talk shows some examples of systems that can take advantage of them (as well as the strengths of FAST and Wikipedia terminology). Whether or not we implement such systems broadly, I hope we continue to support ordered, coordinated subject headings in the underlying data of the next-generation catalogs we build. It’s easy enough to automatically map from such data to FAST term sets, for discovery systems that work best with those, but you can’t reliably and automatically go back the other way and keep the original degree of precision.

In short, I want us to build catalogs that let catalogers and readers do more with metadata, rather than less. I hope that these two talks suggest some ways we can do that, and I’d love to hear from folks who have more ideas, suggestions, or questions.

Going beyond 1922: Finding periodicals with (and without) renewed copyrights

As mass digitization progresses, and as copyright terms grow longer, we now have access to much more literature from 1922 and the years before it than from the years that come soon after it. This graph by Eric Crampton, based on research by Paul Heald, is one illustration of the problem. Copyrights from 1922 and before have expired in the US (where much of the mass digitization to date has been done); copyrights from 1923 or later may or may not still be in force, and research is required to determine whether a book from after 1922 is still in copyright in the US.

As it turns out, many of them are in the public domain, particularly those published in the US between 1923 and 1963, whose copyrights had to be renewed to stay in force. Some organizations, most notably HathiTrust, have opened access to hundreds of thousands of book volumes by researching renewals and other information on books with non-obvious copyright status. In contrast, their open access collections of periodicals and other serials, like those at JSTOR and the Library of Congress, end at 1922.

Why not continue serials past 1922 as well? One reason is that copyright clearance for serials is quite a bit more complicated than for books. Like a book, an issue of a serial can be copyrighted and renewed. But an individual contribution to a serial can also be copyrighted and renewed on its own. So in order to see whether a serial volume is in the public domain, you need to check a lot more potential copyrights. [1]

However, now that all active copyright renewal records are represented online, it’s possible to check whether a particular serial issue or volume is under copyright. It hasn’t been feasible to do it quickly until now, though, because a lot of the Catalog of Copyright Entries, which has records of all copyright renewals, is only online in page-image form, and not as reliably searchable text. But over time I’ve been compiling an inventory of periodicals that have renewals of various sorts. That inventory has now reached the point where you can determine if periodicals have renewed copyrights, and when the first active renewal was made, with a few text searches.

Here’s what you search:

First, look for the name of the periodical in my First Copyright Renewals for Periodicals web page. This will tell you if there were any issue renewals made between 1950 and 1977 (for publications between 1923 and 1950). It will also tell you if there were any contribution renewals made between 1950 and mid-1953 (for publications between 1923 and 1926). I’m still working on adding periodicals whose first contribution renewal was made after mid-1953, but the resources below can be used to find those.
Second, search for the name of the periodical in Project Gutenberg’s compilation of copyright renewals. This is a large text file that includes both book renewals and periodical contribution renewals from mid-1953 to 1977. It doesn’t have periodical issue renewals, but those are covered in the other resources in this list, and the contribution renewals neatly pick up where my file above left off.
Third, search for the name of the periodical in the US Copyright Office online database. This contains renewals both for periodical issues and for periodical contributions (and any other kind of renewal as well) from 1978 onward. So it picks up where my First Copyright Renewals file leaves off for issues, and where Project Gutenberg’s file leaves off for contributions. My “How Can I Tell Whether a Copyright Was Renewed?” FAQ includes instructions on how to search this database.

If you search all three of these (using appropriate keywords and name variants– periodical names can vary over time) and don’t find any mention of the periodical, there’s a good chance there aren’t any copyright renewals you have to worry about.[2] If you search them in the order above and do find mention of the periodical, you’ll have a good idea about when in the periodical’s run you have to start worrying about copyright renewals.

So far, I’ve found out that, just as most periodical issues of the mid-20th century do not have renewed copyrights, most periodical contributions of the mid-20th century don’t, either. I’ve been looking primarily at the 1920s so far, and most of the renewed contributions I’ve seen are fiction. Short stories and serialized novels appeared in a variety of periodicals, including upmarket magazines like Collier’s and the Saturday Evening Post, pulps like Weird Tales and Marriage Stories, and daily newspapers like the Oakland Tribune and Cincinnati Enquirer.

There are a number of nonfiction renewals as well, but these tend to be less common, and often only involve a few people. Many of these renewals are for feature articles or poetry by well-known figures. (Edna St. Vincent Millay, for example, published poems in a number of magazines, and renewed many of their copyrights.) Renewals in scholarly and trade journals are rarer still, and even more strictly limited to particular people. The widow of S. S. Goldwater, for instance, appears to be responsible for the only three active contribution renewals I have found in the Journal of the American Medical Association; she also renewed the articles he wrote for Modern Hospital.

Some periodical copyright renewals may be outliving the periodicals themselves. One contribution renewal I found was for an article in the January 1926 issue of New Eve, a magazine of flapper fashion, fiction, and photography published in New York. I have not been able to find any copy of that issue anywhere. The Internet Archive has a scan of the May 1926 issue; WorldCat knows of two rare book collections that have the April issue. And that’s it– I haven’t turned up any copies of the January issue, or any other issue, in library catalogs or book sites. When that issue enters the public domain, hopefully in a few years, there may be no copies left to scan. Maybe some still exist, but the collections that hold them haven’t been cataloged, or made their catalogs available online. Either way, these records point to useful work that libraries can do to preserve rapidly vanishing cultural resources of the 20th century.

It’s also clear that libraries can make public a lot of the scholarly and professional resources of the early and middle 20th century. Since scholarly and trade journals have a very low renewal rate, both for issues and contributions, and nearly everything published in those journals was original content (so one doesn’t have to worry about reprints from other publications), the content of many journals is completely, or almost completely, in the public domain well past 1922, in many cases as late as the early 1960s. Moreover, in many fields journal articles as much as 50 or 100 years old are still of research or scholarly interest. If libraries open up the content of mid-20th-century journals, scholars and readers could benefit at least as much as they do now from HathiTrust opening up the content of mid-20th-century books.

I’ll be continuing to sweep through periodical contribution renewals from the 1920s onward, and updating my “first periodical contributions” pages as I do so. But even with what I have now, the three-step renewal search I describe above is a powerful way to find out the copyright status of periodicals published in the US. I hope I’ll hear soon about good things libraries and readers are doing with them.

[1] The situation is even worse when copyright terms are based on an author’s lifetime, as is the case in most countries outside the US, and in the US for recent publications. You might need to check the lifespans of hundreds or even thousands of identifiable authors to see if a particular serial volume is in the public domain. (Back)

[2] Fine print caveats: These searches will not find renewals in other categories that might conceivably appear in a periodical, like images, music, and drama. However, there are very few copyright renewals for published images (and are all browsable in the Catalog of Copyright Entries), and music and drama only appear in certain kinds of periodicals. Also, searches on a particular publication might not turn up renewals for copyrighted materials that also appeared in other publications. This is mainly an issue for periodicals that commonly reprinted items, included book excerpts, or that ran syndicated material. Also, it’s possible that there are typos or other mistakes in the files that I or Project Gutenberg prepared, so if you want to minimize risk, you may want to double-check against the page images of catalog registration records, or the original Copyright Office records. Periodicals not published in the US may be exempt from copyright renewal requirements, but as I’ve noted before, many periodicals of non-US origin can be considered published in the US for the purpose of copyright law. Finally, I am not a lawyer; for legal advice, consult with appropriate counsel. (Back)

Some presentations on reading (and linking) libraries

Have you ever wanted to read a library? I know I have. More than once when I was young, and introduced to a new library, I’d contemplate, if just for a moment, whether I could read every book in it. (Eventually I’d do the math and realize I’d never finish in my lifetime. But I’m pretty sure I’m not the only person who thought about doing it.)

While human beings can’t literally read the text of every book in a library past a certain size, “reading libraries” from a distance — that is, understanding the nature of a large collection of information, and how it can answer out questions and satisfy our interests — is something that researchers do a lot, and for good reason. We do it, in different ways, when we survey the research output of an individual or institution; when we scan shelves of books for relevant sources; when we assess the likely worth of material from a given source; and when we conduct searches in catalogs, or library databases, or web search engines, for books, articles and websites that will answer our questions.

Computer-mediated aggregation and analysis of online information gives us a whole new set of methods for “reading” collections from various distances and viewpoints. Collectively, we’re still trying to determine the most useful methods, and the most effective implementations, for “distant reading” of collections of information. Lately I’ve been putting a fair bit of time into thinking and talking about some of them.

Back in August, for instance, Anne Seymour and I gave a talk at the VIVO 2013 Conference in St. Louis, titled “How to Read 200,000 Publications: VIVO and the Intelligent Evaluation of Scholarship”. VIVO is a semantic web application that we’re now using at Penn to aggregate, analyze, and share information about the research publications of thousands of Penn’s faculty researchers in health sciences. In our talk, we discuss ways that people could use the VIVO service and its data to better evaluate the work of these researchers. We emphasize the importance of looking at a body of work in a variety of ways, both from a distance, and close-up. And we note the dangers of relying on tempting shortcuts, like “impact factors“, to substitute for more relevant and considered assessments. We’ve posted the slides and approximate script for our talk online.

Another kind of library-reading occurs where library users ask “What does the library have to offer on a topic I’m interested in?” (“How should I start reading what it offers?” is an important followup question as well.) Librarians expect their catalogs and discovery systems to be able to answer these questions for their patrons. Some of us, like Lorcan Dempsey, say that libraries can provide much better answers if our systems support “full library discovery” encompassing many different types of information. Some, like Dale Askey, say that “Google won the discovery wars years ago“, and conclude that it’s not worth spending a lot of resources on library-specific systems that “are of little interest to our users”.

I see some truth in both of these positions. I believe that libraries have a lot of relevant offerings (not just books, articles, and other information sources, but also guidance, expertise, and space) that can help answer researchers’ questions. I also believe we could do a lot better at making these offerings apparent and intelligible to our users. But I agree that library discovery systems are not going to retake the lead of Google (and other heavily used Internet sites like Wikipedia) as common starting points for research. The findings in OCLC’s recent Perceptions of Libraries survey report, which includes the note on page 32 that “Not a single survey respondent began their information search on a library Web site,” are sobering confirmations of the futility of trying to out-Google Google.

That’s why I’ve lately been stressing the importance of making it easy for library users to find their library’s offerings from wherever they start searching. Search-engine-friendly catalogs and digital repositories are important parts of solving this problem. So is the ability to seamlessly route library users to resources that the library licenses, without them being stopped by a paywall. And so is referring users looking up topics in their favorite search engines and information sites to overviews of what their own libraries offer on those topics. Those overviews could be the attractive “bento box” overviews that Lorcan Dempsey discusses, or they could just be plain old library catalog searches, if that’s all a library can manage. Either way, getting a user to those summaries of a library’s offerings helps ensure they will get used when needed.

OCLC has offered local library referrals for specific bibliographic items from Worldcat.org for a while through its deep linking service, which is based on ISBNs and similar identifiers. More recently, I’ve been implementing local library linking for specific subjects, authors and works through the Forward to Libraries service. I’ve described in previous posts how this service now provides links from The Online Books Page and from Wikipedia, and can link to an ever-growing array of libraries of all kinds. Lots of other sites could be linking out to libraries using the FTL service as well. I’d like to get more digital and local libraries involved both in supporting and improving library-referral links.

To that end, I’ll be giving a couple of talks on Forward to Libraries in the next few weeks. At the DPLAfest in Boston on October 25, I’ll be co-leading a workshop on “Using Digital Tools to Extend the DPLA and Connect with Local Libraries“. Participants will not only see how they can take advantage of Forward to Libraries, but they’ll also get to see John Sarnowski describe and demonstrate tools that local libraries can use to digitize and share their local content with the world. At the Digital Library Federation Forum in Austin on November 5, participants will get to hear about FTL in “Forward to Libraries: Experiences Connecting Digital Libraries, Local Libraries, and Wikipedia“, one of a series of snapshot talks that also features many other interesting projects and ideas.

While my formal presentations at both meetings will be relatively brief, I’m very happy to talk more with interested folks about how they can provide links and overviews for local library resources. And outside those meetings, I’ll be glad to help hook up your library. At this point, all of the general research libraries in DLF should be registered with FTL, and hundreds of other libraries are registered as well. If you’d like to register your own library, or let me know of better ways for FTL to link to your library, just drop me a note via the service’s request form.

50,000 online book and serial catalog records released as CC0

What a difference a few years can make. A few years ago, folks in the library world (myself included) were arguing about whether it was a good idea to let other people copy and build on their catalog records. Whether or not libraries could or should reuse and redistribute records from WorldCat, for example, was up in the air. Some of us were starting to take small steps towards putting catalog records under open licenses. For instance, I licensed the catalog records I created for The Online Books Page under the Creative Commons Attribution-ShareAlike license some years back. At the time, that was farther than many library projects were willing to go.

By now, though, there’s been a definite shift towards wider and more common opening up of bibliographic records. Large libraries like the German National Library and Harvard have released millions of the MARC records into the public domain. OCLC has revised its data policies to give their blessing to member libraries releasing their catalog data under ODC-BY. (They’ve also released some of their own data sets, like VIAF, under that license.) And large online library collaborations like Europeana and the Digital Public Library of America have adopted a policy of public domain status (using the CC0 declaration) for their bibliographic metadata. Both of these projects are now supplying promising platforms for projects that can aggregate, reuse, and build on this data in interesting and useful ways.

Now The Online Books Page is joining the CC0 party as well. Yesterday I put a CC0 notice on the more than 50,000 catalog records I’ve created for The Online Books Page’s curated collection over the last 20 years. (Yes, the site turns 20 this summer– it’s hard for me to believe it’s been up this long.) The “curated collection” refers to all the records that I’ve personally edited, as opposed to automatically importing from other projects. (Those records, automatically imported from sites like HathiTrust and Project Gutenberg, account for well over 1 million more books, and make up what I call the “extended shelves” of the site. I replace extended shelves records with personally edited “curated collection” records on request.)

While 50,000 records isn’t a huge number, compared to the number of records found in massive metadata collections at places like Google Books and the Harvard Library, I think it is still a useful metadata compilation. It covers a lot of important free online books and serials that for one reason or another are not in the large electronic book archives. I’ve also made various efforts to enhance the value of these records beyond what many of the more industrial-scale projects provide, including collating multi-volume works and serials, applying Library of Congress subject headings that support interesting modes of subject browsing, standardizing personal names (which enables linking between libraries and Wikipedia), and various other improvements. And I continue to add new records and improve existing ones in response to reader requests and corrections, and make them CC0 when I do.

More details about what data’s been placed into the public domain, and how to get our data, can be found on the site’s Copyrights and Licenses page. I hope people find it useful. And I’m thankful to all the other people and organizations that are now openly sharing their bibliographic data, and the people that are using it to make works easier to find and use online.

Categorizing people without marginalizing them

Libraries and bookstores have perennially faced the problem of how to organize books on their shelves. There’s a tension between making certain books easy to find for readers with one set of interests, and making them more difficult to find for other readers. For instance, some libraries and bookstores near me have a section for African American fiction. Readers particularly interested in African American authors can easily find their books in this section. But if novels by African American authors are shelved there instead of in the general fiction section, readers browsing general fiction might not find many African American authors there. Similar issues have arisen with genre fiction sections in libraries. A separate “Science Fiction” section can be a convenient service for for fans of that genre. But some readers have objected that such sections push science fiction off into a corner, making it easy for “mainstream” readers to overlook the genre.

In theory, online libraries shouldn’t have as much problem organizing their books and subjects. Freed from the physical constraints of bound paper and shelves, the same book can be placed in many virtual locations, not just one. But in practice, many of the problems of categorization persist in the online world. Last week, for instance, Amanda Filipacchi noted in the New York Times that Wikipedia’s category listing of American novelists was disproportionally male, in part because some editors had been taking women authors out of this category and moving them to the more specialized “American women novelists” category. As far as I can tell, Wikipedia policy does not call for this sort of marginalization, but it doesn’t prevent it from happening either. It’s not just a matter of editors with an agenda and time on their hands; it also happens because manually filing people under multiple categories takes more effort than filing them under one, and it’s easy to neglect or forget to put someone in a broader category after placing them in a narrower one. So people are classified under women authors but not authors, under chemists but not scientists, under Catholics but not Christians. Readers who look for articles in the more general category listings can easily miss people who are only filed in the more specific ones. (And even if those category listings were not originally intended for browsing, many Wikipedia readers do use them that way.)

In systems that have explicit hierarchies of categories (such as Wikipedia categories, or Library of Congress Subject Headings), there’s a fairly straightforward way to solve this particular problem: When a person is placed in a specific category, the system should automatically also place them in any broader categories of people that encompass the original category. If someone is categorized under “Women chemists”, for instance, they should also get automatically categorized under “Chemists”, “Women scientists”, and “Scientists”. This inclusion can be implemented in various ways, but the important thing is that narrowly-classified people should be just as visible for readers browsing the broader categories as people that were explicitly classified under the broader categories.

I implemented this automatic category promotion yesterday on The Online Books Page, which for a while has classified people that are the subject of listed biographies. Consider, for instance, St. Catherine of Siena. The catalog data for her biography in The Online Books Page categorizes her as a Christian woman saint, and readers have been able to find her under that subject for some time now. Thanks to the new algorithm, readers will now also find her when they browse broader subjects like Christian saints, or saints generally.

We could be doing this sort of thing in other library catalogs, and in Wikipedia, as well. Why aren’t we? I’ve seen a few objections to the idea:

It’s too hard to implement? It doesn’t have to be. It took me just part of a Sunday afternoon to implement the feature on The Online Books Page, and I suspect a good programmer who was familiar with (and could modify) the relevant source code would not have much trouble implementing the feature in a well-designed catalog or Wiki. In my experience, I had to spend more time modifying my data than modifying my code. The Library of Congress Subject Headings, the subject system used by The Online Books Page, is not complete or consistent in its subject hierarchies, and I had also miscoded some topical subjects as people. But it’s possible to clean up and enhance this kind of data, and doing so often benefits both present and future applications of the data.

It defeats the purpose of hierarchical categories? I’ve seen this objection made in some of the Wikipedia discussions around this issue, and it doesn’t make sense to me when I think it through. Far from being useless, the category hierarchy is precisely what makes it possible to automatically promote people in narrow categories into broader categories. It also helps save the time of categorizers; they only have to explicitly place people in precise categories, and if the hierarchy is well-constructed the system will automatically take care of the broader categories. (If the system also keeps track of which category assignments are explicit and which are automatic, it can also update them appropriately when categorizations or hierarchies get edited.) I’m also not flattening hierarchies across the board; I’m only recommending at this point that this sort of promotion be done for people, in categories of people. (More generally, it might be useful for any kind of individual instance that is categorized under abstract classes of those instances. But doing it for people is a good start.)

It makes the broader categories too crowded to be useful? In a comprehensive catalog such as Wikipedia, there will be a lot of people in categories like “writers”, once you include all the people in sub-categories. But there still will be a lot of people in that category even if you banish all the women to a “women writers” subcategory. Creating another category for “men writers” doesn’t really solve the problem; all it does is force people to choose which gender they want to browse, instead of letting them browse writers of both genders if that’s what they want to do. And after the split, the broader “writers” category will most likely still be left with a random assortment of writers without gender classification, who might or might not be the people a reader is most interested in.

Well-designed interfaces make it possible to usefully browse large collections of items. Relevance ranking, for instance, can be used to put the most notable examples of a category at the top of a long list of its members. That’s in fact what we routinely expect to happen in good search engines. And mechanisms like faceted navigation (used in many online catalogs) and subject maps (used on The Online Books Page) make it easy to shift focus to more precise or related categories based on a reader’s interests. In systems that implement these features, categories with lots of members are good things to have, not bad things.

I haven’t yet implemented relevance ranking in my subject browsing. Right now, The Online Books Page doesn’t actually classify many people to begin with, so most of my categories don’t have a lot of people in them. But I could see a number of ways to implement such ranking in a catalog like The Online Books Page, or in Wikipedia, which I can discuss later if there’s interest.

In summary, then, well-designed catalogs and wikis should be able to categorize people comprehensively without marginalizing them. Three features that make this possible are:

detailed, well-organized systems of categories and their relationships
systems that automatically show people in broader categories when they’re classified in narrower ones
and ranking and navigation mechanisms that make it easy to pick out the people with the most general interest, or the qualities of interest to a particular researcher, from a large overall set of people.

I’ll continue to work on implementing these features on The Online Books Page, and would be very interested in participating in discussions of how they can better work there, in other catalogs, and in systems like Wikipedia.

Open data’s role in transforming our bibliographic framework (Updated)

I’ve seen a fair bit of buzz of the Library of Congress’s recent announcement of a new initiative for transforming the bibliographic framework of the library community. The announcement notes a number of recent developments that are drivers for such a transformation, including RDA, FRBR, the semantic web of linked data, and a growing consensus that the library community needs to move beyond MARC as its central data standard.

There’s another very important driver not explicitly mentioned in the announcement: the rise of open library data. More and more bibliographic and other library-related data is now freely available and reusable online, and it enables all kinds of improvements in resource discovery. I’ve previously discussed how open data for Library of Congress Subject Headings and Hathi Trust online books allows me to manage a catalog of over a million books, create rich subject browsing interfaces, and improve the quality of library catalogs at Penn. OCLC has also compiled data on millions of authors in VIAF, and now makes it available as open data. I recently downloaded their data set, and hope to get at least as much benefit out of is as I have with subject data to date. (I hope to report on early experiments before long.)

And there’s more big news from OCLC. At last month’s Global Council meeting, Karen Calhoun gave a presentation saying they were considering releasing letting members release WorldCat data under an open license. I’m very excited to see this development; as I said a couple of years ago (in a panel discussion that also featured Karen Calhoun) it would make a huge trove of the library community’s bibliographic intelligence available to exploit and extend in all kinds of ways that could make it easier for library patrons to discover information resources. This is no small step for OCLC– at the time of our panel, OCLC was concerned that opening access to WorldCat data might threaten the sustainability of the WorldCat cooperative. If they’re seriously considering opening their data now, I suspect that a good number of WorldCat members have indicated that it’s an important and worthwhile thing to do, and that the benefits outweigh the risks.

If you also see potential in open library data, now is an excellent time to join in the discussions that the Library of Congress and OCLC are inviting. The more these and other leading organizations in the library community see how open data can advance the goals of the community, and how open data initiatives can get the support needed to be sustainable, the richer the knowledge base that our evolving bibliographic framework will support.

I don’t yet know where and how the LC and OCLC conversations will be taking place. But I’d love to hear readers’ thoughts and pointers about using and supporting open bibliographic data in the comments here.

Updated May 27 to clarify: I’ve heard from Roy Tennant that the OCLC proposal under consideration refers to WorldCat members releasing “their catalog data containing WorldCat records”, not OCLC itself releasing data (as my post originally stated). This is still potentially a big deal, even if it doesn’t mean opening up all of WorldCat. I’ve updated the post, marking edits with strikethrough and bold, and will talk about this more in the comment thread.