Each publication should be assigned a unique ID (UID). This is inspired by the integration of many different applications that is enabled by the concept of a citekey in BibTeX. APIs should enable users to submit UID and receive metadata for any publication (whether in JSON or BibTeX, whether strictly citation info or also social info about tags, other users, links etc). There should also be a number of ways to determine a publication’s UID through various lookups.

There are (roughly) two choices for the format of a UID. The first would be a randomly generated (or sequential) ID with no semantic meaning, whether with numbers or letters etc. The second would be the citekey format which researchr currently uses. The advantage with this is that it is familiar to users (of LaTeX / researchr etc), and immediately conveys some minimal information about a citation. Through use, certain frequent citations might even be recalled actively or passively. Certainly, it is much easier to reorder three publications cited in a blog post using citekeys (I’ll put the scardamalia2006knowledge first, and then mention johsnson2000corruption) than using random IDs (See for example 3093049, 304955 and 88585).

However, there are a few challenges with using the citekey format. The first is generation and the second is collisions. Although the general principle is well understood (last name of first author + year + first word of title) there is a number of permutations, for example

• I prefer manually changing van2006knowledge to vanderwende2006knowledge

• what to do with punctuation, is it peter2006knowledge or peter2006knowledge-integration

• it often makes sense to include the first word with more than n (=3?) letters, etc.

This results in citekeys generated by researchr or other tools (Google Scholar) and Scrobblr to be different. Some of these we can just define arbitrarily, but we might want some decent algorithm to solve the first point above - perhaps joining the words of the last name without spaces.

Given that we can thus generate nice citekeys from submitted metadata (much of which won’t even have a citekey, or have a citekey in a totally different format), we encounter the problem that the citekey in the database might differ from the citekey in the user’s local system. One approach would be to use Researchr or other plugins to “harmonize” these (i.e.. automatically modify citekeys on the user’s end) - this would have to be done early in the import process, because everything locally is tied to the citekey (PDF name, wiki pages). (Of course, in the future Scrobblr will be the first place we go to download papers in our fields anyway so theoretically we won’t even have this problem :)) Or we could just accept that there will be a discrepancy here.

The second problem however will be collision. It is likely that there will be cases of several papers generating the same citekey. Again we’ll need a way of resolving this. A simple way would be to add “b” to the year or something like that - not very elegant, since it will look kind of “random” when viewing it outside of a context. Another approach could have been to go back and give both articles a longer citekey to avoid collision (perhaps the first two words of the title), however, given that a citekey once assigned should be absolute, this is impossible.

Given that we can solve all of these things, the final concern is user confusion about local citekeys and Scrobblr citekeys, given that they look so similar. One way to mitigate this in practice would be to come up with some notation for linking to citekeys which specified that they were Scrobblr citekeys. Currently we are using citekey for citations, but this is purely random, it could easily be something else. It would however be great if it was something both easy to type, easy on the eyes, and still fairly unambiguous. Since citekey is rarely used on the web today, it would for example be easy to write a plugin that scanned a blog post for this notation and recognized citations. (Not a great example, since the citation would probably be transformed to a full citation by the blog post software anywhere, but…)

There are two possible ways to populate the Scrobblr database, the first is user contributions, and the second is “canonical contributions”. I will start with the second one, because while it might be much rarer in the beginning, it avoids a number of problems with the first category.

Despite all the effort users go through in typing in and reformatting citations, “perfect” citation entries are out there, at least in most cases. The metadata put out by a publisher of a given article or chapter should in most cases be considered “canonical”, and since almost all publishers maintain these in a machine-readable format internally, theoretically it should be easy to integrate these. In practice, the situation is of course different.

However, there is a number of possible approaches - how attractive these are depends partly on how specialized we intend to keep Scrobblr (or to put it in other words, how much work we want to put into acquiring and hosting metadata for publications which we ourselves are unlikely to be interested in). There are a few categories of canonical or mostly-canonical sources that we could with some work integrate:

• large pre-print archives that enable bulk download, for example (only?) arxiv.org which enables full download both of their entire full-text article collection, and metadata in BibTeX (740,000 publications)

• repositories which provide standardized APIs, such as most of the institutional repositories through Directory of Open Access Repositories (DOAR)

• Directory of Open Access Journals provides the TOCs of 4000 OA journals (has API)

etc. In addition to having ideally mostly correct metadata, the sources above all provide links to the full-text of the journal articles, which means that we can also extract PDF hashes.

The obvious way for users to contribute is of course to use researchr, and we already have fully functional demo of submitting both metadata and PDF hashes. However, we would also like users of other platforms be able to contribute. Submission of metadata is quite easy - it is no problem to tie into Mendeley API for example, to import your entire library or selected publications, the same for many other websites such as CiteULike. The GScholar bookmarklet is also an attractive solution for ad-hoc imports. Finally, one could imagine a web interface for directly entering citations into Scrobblr.

A common problem with many of these methods (apart from the researchr one), is that it will be difficult or impossible to access the PDF and generate a PDF hash. In the case of Google Scholar, we can do this if there is a PDF link present, in the case of Mendeley and Papers, we might be able to do this with an desktop plugin, but not with a purely online API.

There are

Separate from the concept of people submitting incomplete publications is the problem of people submitting completely erroneous information, whether intentional (as an effort to spam) or unintentional (a script gone mad that fills up the database with 1000s of spurious entries). Ideally the use of an API key (and we might need a captcha for sign-up etc), as well as a non-standard API, would mitigate most automatic attempts. We could eventually have a way of flagging erroneous entries, and if enough from the same user were flagged, all his/her contributions could be removed automatically. (Of course, if someone were really determined to undermine the site specifically, creating multiple accounts etc, that would be hard to defend against - not something to worry about in the short-term though).

The problem of “unintentional spamming”, by users creating thousands of entries through erroneous scripts or experimentation is more difficult to secure against. There might be an option in a user’s control panel to view last import, and cancel the entire last import if something went wrong.

The much more common problem will be people submitting incomplete citations or slightly different citations. This is tightly connected to duplication-detection (below).