Citing AI in the New Bluebook
Tackling the most important emerging issues at the intersection of AI and law.
The venerable Bluebook has just released its twenty-second edition. Among the many innovations this new edition brings is a standard for citing AI-Generated Content: Rule 18.3.
While standardizing citation conventions for AI-Generated Content is indeed a timely and valuable endeavor, I’m afraid the Bluebook editors have fallen a fair bit short in the substance and structure of Rule 18.3
Overall Structure of Rule 18.3
Rule 18.3 is split into three parts:
Rule 18.3(a): “Large language models”
Rule 18.3(b): “Search results”
Rule 18.3(c): “AI-generated content”
You can already see the problems mounting.
Rule 18.3(a) vs. 18.3(c)
First of all, what is supposed to be the difference between “large-language models” (Rule 18.3(a)) and “AI-generated content” (Rule 18.3(c))? Content generated by LLMs is a type of AI-generated content!
Examining the rule more closely, one way to interpret Rule 18.3(c) is that the editors want you to use it for AI-generated content other than text. That is, Rule 18.3(c) is to be used for things like images, audio recordings, and sound. The Bluebook offers the following examples of materials to be cited under Rule 18.3(c):
Pablo Xavier, Photograph of Pope Francis in a Puffer Jacket, in This is Not a Real Photo of the Pope in a Puffy Coat, Snopes (Mar. 26, 2023), https://www.snopes.com/fact-check/not-real-photo-pope-in-puffy-coat/ ;[https://perma.cc/3EN7-BP4P] (generated by Midjourney AI).
Illustration of a Tornado on the Moon (on file with the Columbia Law Review) (generated by DALL-E 3).
Both of these are formatted as Rule 18.9: Images. The same would presumably be true of AI-generated videos (Rule 18.7: Videographic Material) and audio (Rule 18.8: Audio Recordings and Streaming).
An alternative interpretation, however, is that you should use Rule 18.3(c) instead of Rule 18.3(a) when an alternative citation format for the content is available. Since there is an existing citation format for images, Rule 18.9 applies. This interpretation is perhaps supported by the text of the Rule, which says “Content that has been generated by AI should be cited according to the relevant Bluebook subrule . . . .” This interpretation also makes sense when considering the text of Rule 18.3(a), which concerns the “prompted responses of large language models,” implying a concern with the type of typically unpublished chatbot conversations that might not be easily citable under other rules. Adding further weight to this interpretation, Rule 18.3(a) also gives as an example citation:
Luke Cronin, Google Gemini Advanced, “Who would make a better Supreme Court Justice: Beyoncé or Taylor Swift?” (Mar. 29, 2024) (on file with the Columbia Law Review).
This interpretation would imply that, if an entire journal article was generated by an LLM, it should be cited as an article under Rule 18.3(c), not 18.3(a).
Regardless, subsequent editions should clarify the difference between Rules 18.3(a) and (c).
Why is 18.3(b) here?
Rule 18.3(b): “Search results” seems misplaced under Rule 18.3. While of course modern search engines frequently incorporate AI-generated elements (e.g., summaries), there is no indication that Rule 18.3(b) is uniquely concerned with that, and the examples given are not especially AI-forward:
Bing, “The Bluebook”, 6,050,000 results (May 22, 2024) (on file with the Harvard Law Review).
Westlaw, +“Disclosure Controls” +“SEC”, 1 result (Mar. 30, 2024) (on file with the Yale Law Journal) (filtered by “Cases”, “2d Cir.”).
Perhaps they mean “AI-generated” in the sense that search algorithms like PageRank are “AI” and their results are “generated.” Both seem like a stretch.
Other Issues with Rule 18.3(a)
Internally, Rule 18.3(a) has a few issues.
Its first paragraph contains the following sentence:
The tailored nature of deep neural networks individualizes the internet such that a URL will no longer suffice for an increasing number of web-based resources.
I genuinely do not know what this means. What is the “tailored nature of deep neural networks”? Are they really the driving force behind the “individualization of the internet”? Haven’t we been dealing with non-deterministic internet pages for a long time? Seems like a non-sequitur.
My best guess of what this means is that they don’t want you to use the method of parenthetically describing what you did, as in the Westlaw example above. As Rule 18.2(d) says:
If the root URL is used and the site’s format is not clear from the rest of the citation, a clarifying parenthetical should be added to explain how to access the specific information to which the citation refers . . .
They are correct that this is not a good way to deal with AI-generated content. But it’s a problem for non-deterministic processes generally, not AI. It is confusing to include that aside under Rule 18.3(a); they should instead include it as a qualifier under Rule 18.2(d).
Second, the Rule requires “the exact text of the prompt submission in quotation marks.” See the Cronin example above.
There are a few problems with this. First, it neglects the fact that LLM-generated outputs can be the result of multiple rounds of prompting, generation, and refinement: prior outputs form the context for inputs. Presumably, the Bluebook does not that quotation to be something like “Please make the language in that more formal.” But the rule as written requires “the exact text of the prompt submission”; it does not allow for prior context.
Nor, textually, does it allow for the description or abbreviation of the prompt. Prompts can be very long; the best prompts often are. If a prompt is paragraphs long, does the Bluebook really require you to include all of it? Textually, yes, but this is absurd.
Indeed, the Bluebook editors cannot even bring themselves to be consistent on this point. In its overview in Rule 18.1, they include the following example for Rule 18.3(a):
Akesh Shah, OpenAI GPT 3.5, Constitutional Confiscatory Takings (Jan. 15, 2024) (on file with the Columbia Law Review).
This violates the “exact text of the prompt submission” rule. To the extent that the good Mr. Shah’s work is something like an LLM-generated unpublished manuscript (which would be a good reason not to use the “exact text of the prompt” rule, since it would probably be too long and/or multishot), it also throws a wrench into my theory about the difference between 18.3(a) and (c): this seems like it should have been cited as an unpublished manuscript under Rule 17.2.1, as modified by Rule 18.3(a), as follows:
Akesh Shah, Constitutional Confiscatory Takings (Jan. 15, 2024) (unpublished manuscript) (on file with the Columbia Law Review) (generated by GPT-3.5).
(Notice also the correction of “OpenAI GPT 3.5” to “GPT-3.5.”)
The editors deserve credit, however, for the rule requiring a PDF of the generation being cited. This makes sense for the preservation of information, given that we can’t bank on the reliability of AI sharing links forever. This rule could be further improved, however, by requiring or encouraging the PDF to be stored using something permanently publicly accessible, like perma.cc. Authors should also be encouraged to provide, in their PDF, further information that may influence the outputs in question, such as any global prompts and full context.
A Few Other Nitpicks
“ChatGPT and CoPilot” are not “large language models”; they are AI services powered by LLMs.
The rule is ambiguous as to whether the name of the model should include the name of the company that made the model, even when the company does not normally include its name in the model name. The examples are ambiguous: the Shah example above includes “OpenAI”, but the Illustration of a Tornado on the Moon example does not.
Rule 18.3(a) requires the date of the generation and prompt used, but rule 18.3(c) does not. It is not clear why it’s more relevant for the former than the latter.
Concluding Thoughts
I don’t mean to be too harsh on the editors, whom I commend for tackling this issue head-on. But this Rule lacks the typical precision for which the Bluebook is (in)famous. The good news is that the next issue will likely be entirely AI-written and lack any of the lacunae that mere mortal drafters are doomed to leave.
