A few weeks ago, I was contacted by Tim Spalding of LibraryThing, asking if I was serious about the twitter conversation we had:
@griffey fine I’ll pay you to pay others, but now it’s on Twitter…
— Tim Spalding (@librarythingtim) January 15, 2015
He wanted some form of independent analysis of recommendations coming from LibraryThing and from Bookish in Bibliocommons catalogs. After exchanging a few emails, we came to the following agreement:
- I would solicit and select 4 testers for the recommendation systems
- The reviewers wouldn’t be told who had arranged for the review, nor who was paying them (reviewers were offered $200 for the effort) until after they had turned in the reviews and they were posted here
- The reviewers would examine 7 different Bibliocommons catalogs and write up their thoughts, but there was no suggesting of the books they were to search for or anything like that…they had free reign to use the catalogs in questions to look for any books they wished.
- Tim wouldn’t know who they were during the review, and not afterwards until the reviews were collected and this post went live
- There would be no editing on his part of the reviews…what they said, they said. No picking and choosing to rose color anything.
- I would collect and edit the reviews only in as much as needed to anonymize them.
- Tim would get the names of the reviewers after this is posted in order to pay them, but he would have no way of knowing who wrote what among the reviews.
This was as good a protocol as I could put together quickly for a blinded analysis of the two systems. I can promise that as I write this, Tim has no idea who the reviewers were, nor has he edited the reviews below in any way. They are as independent a look at the two systems as I could arrange.
So what is posted below are 4 comparisons, in the voices of the reviewers, between the recommendations provided by LibraryThing and Bookish.
This was done as a consulting job, so in the interest of full disclosure I am also receiving a small payment for my work in doing this.
Now that these are public, I will be sharing the identities in both directions of this endeavor. Who wrote which review will remain anonymous, but both sides will be alerted in order to settle the accounts in question.