Jonathan: I’d be happy to consider a code4lib article if there’s interest. For now, here are my quick answers to your questions:

* It’s not powered by either solr or a dbms. The code is based on a set of Perl modules I started writing back in the 20th century; the data’s kept in something much like Berkeley DB files, but much less space-hungry. I like the idea of moving it to Lucene, which might include Solr, but shifting things over to Java would be nontrivial, Perl versions of Lucene had performance issues last I checked, and using an external Solr instance might or might not improve things (haven’t thought much about this last option; it would at the very least involve more overhead keeping everything running).

* The NT relationships are determined from the authority file encodings plus my heuristic rules. Some of these rules also look at subject assignments in the collection to be indexed. The exact details are a bit long for a comment but I could detail them elsewhere.

* My current plan is to reprocess when LCSH updates, but that’s not a big deal for me. New RDFs seem to come out every month or two. Preliminary processing to prepare the new RDF for my indexes takes less than 5 minutes for the entire RDF. I can then regenerate not only the subject maps but also all other indexes for my collection in another 5 minutes; this is typically done daily. The time required scales up somewhat with collection size, but earlier test indexes with Penn’s whole catalog (and a smaller set of authorities and heuristics, granted) took under an hour, IIRC.

* I do plan to bring in local authorities as well; I would have done it by now except for some character encoding issues. I could see some possible issues with links to obsolete forms, but there should be ways to address this. (Actually, this sort of analysis could also be tweaked to flag obsolete subject terms in one’s catalog, and possibly change them in bulk as well.)

* Yes, I’d be happy to share code, but see comments above about 20th-century Perl. Not sure how useful it will be in general (and I’m not in a position to provide support for it), but I can show it to folks who are interested.

Hope this answers your questions; and again, if folks would be interested in learning more about various technical aspects, I’d be happy to consider doing a paper.

Is it powered by Solr, an rdbms, both, neither?

How do you determine the narrower term relationships (and other relationships), just from the actual authority file encodings, or also using your own heuristic rules? Sounds like some of your own heuristics too, would be interested in learning more about them.

When you get new updates from LCSH, do you have to “re-index” or “re-process” your entire db, even for just minor updates?

Did you consider trying to combine the LCSH authorities and your local authorities, to get a superset of both?

Are you interested in sharing your code, and what language is it written in?

And finally, would you like to write a Code4Lib Journal article about this? I think it’d be a good one!

One thing that LC hasn’t publicized a whole lot is that the loading history for the data can be found here:

http://id.loc.gov/authorities/loads/

In addition there is a Atom feed that makes all the create, update and delete activity available:

http://id.loc.gov/authorities/feed/

The idea with the Atom feed is to allow you to keep your own local version of the records synchronized with id.loc.gov. You can follow the “next” URLs in the atom:link elements to drill backwards until you’ve seen a change at a record update time you already knew about. It’s a bit of an experimental more mainstream alternative to oai-pmh, so I’d be interested in any feedback you might have if it is of interest.