MajinBook: An Open Catalogue of Digitally Mediated World Literature

Introduction

The use of text collections as a basis for linguistic inquiry has a long history that far predates modern computing. However, it was the computational turn of the late 1950s that established this practice as a fully-fledged academic discipline, later known as corpus linguistics. Dependent on technology capable of managing large quantities of machine-readable text (McEnery and Hardie), early corpus linguistics often aimed to study the building blocks of language (e.g., grammar and lexis). With the increasing availability of computational power, other subfields soon broadened the practice’s aspirations. Sociolinguistics and discourse analysis are prominent illustrations of this methodological borrowing, applying corpus techniques to the study of social structures and norms. Similarly, the field of distant reading applies these methods to uncover patterns in literary history. In this context, the recent progression of Computational Social Science (CSS) can be seen as an advance in scale (i.e., utilising millions of books and social media data) and in tools (i.e., applying advanced machine learning and network analysis) rather than a fundamental change in the long-standing goal of understanding culture through texts.

The vision of using extensive book corpora to represent culture at a large scale was significantly advanced by mass digitisation projects. Beginning in 2002, the scanning effort Google undertook culminated in the Google Books project and yielded, among other resources, the dataset for a foundational “quantitative analysis of culture” (Michel et al.). Shortly thereafter, the HathiTrust Digital Library (Christenson) was formed as a partnership between Google and major research libraries. Now containing over 18 million volumes, HathiTrust has become a go-to resource for supporting broad claims about culture through the computational analysis of books. Its utility is demonstrated by its application across diverse fields, including the history of science (Murdock et al.), literary history (Underwood), gender studies (Underwood et al.), and musicology (Downie et al.).

Despite its scale and utility, HathiTrust is not without significant limitations and biases, many of which stem from its origins. The collection itself is not a neutral representation of world literature; its composition privileges large research universities, while effectively excluding smaller, nonaffiliated institutions and other forms of participation. Furthermore, as a collection built from scanned physical books, the quality of the underlying text can be inconsistent due to data integrity issues and errors from the Optical Character Recognition (OCR) process. The most significant challenge, however, relates to access. The “dark history” (Centivany) of HathiTrust’s formation was fraught with legal and political tensions surrounding copyright law. Consequently, a large portion of the collection containing in-copyright works remains inaccessible for direct reading. This forces researchers to operate within the controlled environment of their research center’s secure “data capsule,” which not only presents a steeper learning curve but can also inhibit the broader, collaborative evolution of new computational tools and methods that thrive on open data. At the time of writing, the HathiTrust Research Center, which operates this data capsule environment, is scheduled to lose its funding at the end of 2026, with no confirmed successor for computational access to in-copyright materials. These combined issues of representational bias, data quality, and the constraints of a closed or even closing research environment highlight the challenges of using even the largest institutional corpora to study culture.

These limitations have given rise to a new frontier for corpus-based research: the large-scale “shadow libraries” that operate outside of formal institutions. Platforms like Library Genesis (LibGen), Sci-Hub, and Z-Library have emerged as a direct response to the access problem, aggregating tens of millions of books and articles in defiance of paywalls and copyright restrictions. These are not merely illicit collections; they are vast, user-driven archives whose very existence represents a form of crowd-sourced cultural curation, making them an essential new source of data for the computational social sciences and humanities (Karaganis). The recent development of Large Language Models (LLMs) has opened Pandora’s box in this regard, moving the use of these datasets from a niche practice to a central component of modern AI. Major companies have publicly acknowledged using such sources in academic papers (Lu et al.) and court cases (Kadrey et al. v. Meta Platforms, Inc.; Bartz et al. v. Anthropic PBC), where their legal defence of “fair use” has already been partially upheld.

The rationale for our study derives from this context. While these new data sources have been processed as bulk, undifferentiated training data, their use has yet to permeate CSS research. The poor quality of shadow libraries’ metadata hinders the precise identification and sampling of content. Such precision is essential for representing a period, a style, or literary culture as a whole, and for navigating between these abstractions and their precise instances, that is, clearly identified books.

To bridge this gap, this paper introduces MajinBook, an open catalogue designed to resolve this metadata challenge and unlock the full potential of these vast literary archives for cultural analytics. Our method leverages metadata from both shadow libraries and a social reading platform to construct a cohesive bibliographic scaffold. This scaffold binds available editions to their original work and first publication date, resulting in a high-precision corpus of over half-a-million books spanning three centuries, along with secondary datasets to foster future works.

This paper is organised as follows. First, we introduce our data sources and their initial processing, namely the shadow libraries LibGen and Z-Library, and the social network Goodreads. We then detail our linkage strategy, its evaluation, and its outcome. Finally, we conclude our paper with some legal considerations about our endeavour.

Shadow Libraries

Discarding PDFs

The PDF format is highly varied, comprising everything from partial, amateur scans to well-indexed official publisher versions. Consequently, consistently parsing this content to extract clean, raw, integral text remains a significant technical challenge. Notable pitfalls include OCR discrepancies and the difficulty of preserving the original content’s reading order. E-books, on the other hand, are natively digital structures, much like webpages, making their content and metadata entirely machine-readable. To include PDF content would be to place items of potentially dubious quality on a par with the very issues found in other scanned corpora, such as HathiTrust, thereby undermining our core objective of a high-quality, CSS-oriented, and natively digital catalogue.

For these reasons, we made the methodological decision to discard all PDFs. This decision introduces a significant temporal bias, favouring recent publications and older works deemed commercially viable enough to be reissued in a modern digital format. Figure 1a clearly illustrates this. On this semi-logarithmic plot, the PDF distribution’s growth is relatively linear, indicating a stable exponential increase over time. In sharp contrast, the EPUB subset exhibits a convex shape, signalling a super-exponential growth rate that accelerates towards the present.

Figure 1: Temporal distributions and biases of key corpora. The figures illustrate the distinct temporal biases of the key corpora, justifying our methodological focus on natively digital content. All three plots are semi-logarithmic (log y-axis), displaying item counts binned by publication decade. (a) Compares the EPUB and PDF subsets of shadow libraries. (b) Contrasts the scanned HathiTrust corpus with all Goodreads editions. (c) Compares our final MajinBook primary corpus (English) to its Goodreads works scaffold. The plots reveal a fundamental difference in corpus structure. The scanned corpora (PDF, HathiTrust) show relatively stable exponential growth (a linear shape), while the social and natively digital corpora (EPUB, Goodreads, MajinBook) exhibit super-exponential growth (a convex shape) accelerating towards the present. This validates our decision to discard PDFs and confirms that MajinBook (c) is a representative temporal sample of its source.

We argue, however, that this skew is not a simple limitation but a deliberate methodological filter. Rather than a bias against the full shadow library corpus, our choice acts as a productive sieve for a specific, n atively digital representation of culture. The trade-off is explicit: We sacrifice the historical completeness of scanned corpora for a dataset of cleaner, more structured, and machine-readable content. Moreover, this productive sieve defines our object of s tudy. We are not claiming to represent “world literature” in its historical entirety, but instead the digitally mediated canon: culture as it is curated, circulated, and consumed in the twenty-first c entury. This temporal bias towards natively digital and commercially viable reissues is not necessarily a flaw, but rather a defining characteristic of our corpus.

This contemporary constraint yields a significant and somewhat counterintuitive benefit. As Table 1 shows, the EPUB subset is far more linguistically diversified (Herfindahl Index (HI) = 0.24) than its PDF counterpart (HI = 0.74). Notably, it is also less concentrated than the HathiTrust corpus (HI = 0.32), demonstrating that our sieve produces a dataset that is both high-quality and linguistically diverse. This resulting language fragmentation, which reduces the share of English from over 85% in the PDF set to only 48% in our corpus (Table 1), can be seen as a positive indicator of cultural breadth, enhancing the dataset’s representativeness for cultural analytics.

Goodreads

Goodreads, founded by Otis Chandler and Elizabeth Khuri, is a prominent social reading platform, launched in 2006 and acquired by Amazon in 2013. It boasts over 150 million members who rate, review, catalogue, and discuss books, creating a vast digital archive of contemporary reader responses and amateur criticism. Academics utilise this extensive dataset to analyse reading activity at scale, compare modern literary reception with historical patterns, and understand how works gain or lose popularity (Walsh and Antoniak; Antoniak et al.; Bourrier; Kousha et al.; Hu et al.). Nevertheless, these studies acknowledge limitations, including demographic biases within the user base (predominantly white, female, and U.S.-centric), and the potential for review manipulation.

We first detail our data requirements and the rationale for our choice of source, before describing our crawling methodology and its outcome.

The Crawl of Goodreads

To harvest data from goodreads.com, we gathered 1,225,390 editions to serve as seeds, the initial sets of items from which our crawl expanded outward. These were drawn from the platform’s various public book aggregations: 30 tags, 642 shelves, 1,419 genres, 8,194 awards, and 30,497 lists. Together, these seed editions corresponded to 990,010 distinct works by 927,707 authors. This initial set of works comprised a total of 11,319,891 editions.

We then recursively expanded this baseline by collecting works from the platform’s recommendations (such as those featured in the “Readers also enjoyed” section), as well as other relevant works by the authors already gathered. To mitigate the long tail of obscure items, we applied a specific inclusion threshold for an author’s additional works: They were required to have at least one rating and two editions. We refer to each successive round of this expansion as a depth, with the seed set at depth zero.

This iterative crawl continued until the author-led discovery of new items plateaued at depth four and ceased at depth five, coinciding with the complete exhaustion of the recommendation-led crawl. These simultaneous terminations suggest that our dataset is a comprehensive representation of Goodreads’s discoverable content

This approach effectively neutralises the risk of algorithmic selection bias. By exhausting the network, we eliminate the path dependency of the recommender system, yielding a corpus defined by the platform’s boundaries instead of its suggestions.

The final dataset comprises 4,778,124 works, 28,105,913 editions, and 2,150,522 authors (cf. Figure 2). In total, 15% of works were discovered via recommendations, 20% came from the seed set, and 65% were retrieved through author-based expansion.

Figure 2: The crawl of Goodreads: Item acquisition and recommendation decay. The figure illustrates the efficiency of our crawl me thodology. The bars show the cumulative counts of editions, works, and authors (left axis, in millions) gathered at each stage. The line plot tracks the number of new recommendations (right axis, in thousands) discovered at each depth. The plot reveals a power-law distribution: The initial depths rapidly capture the most prominent items, while subsequent depths explore a long tail of less-connected content.

To the best of our knowledge, and based on our review of the state of the art (Wan and McAuley; Wan et al.; Kaggle; BrightData; Hu et al.), this represents one of the largest publicly available crawls of Goodreads data published to date.

MajinBook

Matching and Evaluation

Our strategy for matching shadow library clusters with Goodreads works relied on identifier overlap. A given cluster was linked to a given work if the set of all identifiers found within that cluster’s items had a non-empty intersection with the set of all identifiers aggregated from that work’s editions. Any such match flagged the whole cluster as a potential candidate for that work, tagged with the cluster’s language. This process enabled us to link 770,840 Goodreads works to at least one cluster. In total, these retained clusters comprise 4,216,400 shadow library items.

Deeming identifier-based links to be a necessary but not sufficient condition, we implemented a second verification step using metadata. This step performs an exhaustive title comparison for each candidate. For a given cluster and a potential work, we compared every title within the shadow library cluster against every edition title for that work (in the cluster’s language). For each of these pairs, we computed a partial ratio fuzzy match, yielding a score from 0 to 100. Finally, we calculated the mean of this entire comparison matrix to produce a single, robust title score for the candidate. Overall, the distribution of title scores is highly left-skewed, with a median score of 99.1 (IQR = 86.6–100.0). This indicates that the vast majority of identifier-based matches are confirmed by our title matching method.

We then conducted a human evaluation study to validate our matching method and determine an optimal title score threshold. For this, we sampled 200 English-language candidates, using a stratified approach that deliberately overrepresented items with lower, more ambiguous scores.⁴ We built a simple web interface for this task, which, for each candidate, displayed the corresponding Goodreads title and authors alongside the first 100 paragraphs of a random item from the shadow library cluster.

Evaluators from our lab were then asked to assess each candidate and assign one of three labels: “Yes” (a correct match), “No” (an incorrect match), or “I don’t know/I can’t tell.” We concluded the evaluation once every item in the sample had received at least one review. Figure 3 shows the results for the 143 items for which evaluators could come to a conclusion and plots the resulting precision, recall, and dataset size as a function of the title score threshold. The figure clearly illustrates the classic trade-off: As the threshold increases, precision rises, while recall and dataset size fall. Based on our stated goal of a high-precision catalogue, and supported by this evaluation, we selected a final threshold of 80, which achieves a precision of nearly 1.0 (Figure 3).

Figure 3: Precision and recall trade off against each other as the title score threshold varies. The plot shows the point estimates for precision (dashed line) and recall (dotted line), along with their 95% confidence intervals (shaded areas), derived from bootstrap resampling of 143 human evaluations. The solid black line indicates the percentage of the dataset retained at each threshold. A vertical line marks our chosen operational threshold of 80, which prioritises high precision for the final catalogue.

The 57 items (28.5%) that evaluators labelled “I don’t know/I can’t tell” were not factored into the primary precision-recall computation. Our lab colleagues did not determine these items to be incorrect, but rather the items were not able to be evaluated. They typically lacked title pages and started in media res, offering few visual cues for confirmation. To determine whether a high title score could be trusted on these ambiguous files, we conducted a deeper, secondary analysis on the 24 ambiguous items with a title score above our operational threshold (80) and concluded that 22 were correct matches (91.7% precision). The two errors were not random: One was a single book matched to a three-book set, and the other was a different book by the right author. This demonstrates that the title score’s predictive power is robust even for files that are visually ambiguous. It confirms that the formatting issue is statistically independent of the score’s relevance and validates our decision to proceed with the threshold derived from the 143 conclusively-labelled items.

Release

Primary Dataset: The MajinBook Catalogue

We only evaluated our matching methodology on English titles, and 82.9% of our retained matches are for English-language content. This English subset, numbering 539,530 items, therefore constitutes our primary dataset, fulfilling the large-scale, high-quality ambition of our study.

Each entry in this final catalogue includes the following metadata fields:

• Goodreads work ID

• First publication year

• Authors’ full names and Goodreads IDs

• Title

• Rating and number of ratings

• Shadow libraries IDs corresponding to this work in English

Additionally, 84.04% of the catalogue features a list of genres and 99.23% includes the number of reviews.

Secondary Datasets

Despite the overwhelming dominance of English, our methodology captured significant volumes of content in other languages with robust temporal distributions (Figure 4). Although we did not conduct human evaluation for these languages, their title score distributions are highly left-skewed, closely mirroring the distribution of our validated English set.

Figure 4: The primary (English) and secondary datasets differ in their temporal distributions.

While this similarity is a promising heuristic, it is not a substitute for formal validation. Based on these two criteria (significant volume and a comparable title score distribution) we selected three of the largest non-English corpora for release: French (47,960 items), German (35,559), and Spanish (30,169). The sharp drop in volume relative to the source shadow libraries is explained by the fact that our dataset is constrained by the linguistic distribution of Goodreads (cf. Table 1).

The feature coverage for these datasets is identical to the primary English corpus, with the sole exception of genres, which falls to 77.4% for Spanish, 75.5% for German, and 70.1% for French. We hypothesise that this is not an artefact of our matching process but rather reflects the sparser metadata available for non-Anglophone works within Goodreads itself.

We must, however, stress that the precise quality of these matches remains unverified. We release these secondary catalogues as experimental datasets to foster future work, noting that they do not carry the same high-precision guarantee as our primary English corpus.

First Publication Year (10-year bin)

Underlying Datasets

Finally, in addition to the primary catalogues, we are releasing the underlying datasets that enabled this study, namely

• the formatted metadata for the retained elements from the shadow libraries EPUB subset;

• the formatted metadata extracted for every work, edition, and author from the Goodreads crawl;

• the MinHash signatures for every item considered in the shadow libraries.

Legal and Ethical Considerations

The legality of using shadow libraries for research remains a contested and evolving issue, as illustrated by recent lawsuits involving major AI companies (Kadrey et al. v. Meta Platforms, Inc.; Bartz et al. v. Anthropic PBC). At the time of writing this paper, legal frameworks in many countries are being adjusted to find a balance between copyright law and the AI use of copyrighted materials (Sag and Yu). Our project’s legal standing rests on a critical distinction between our final product (a public, metadata-only catalogue) and our process (acquiring and analysing data from Goodreads and shadow libraries).

MajinBook’s final output is a bibliographic index. The titles, author names, publication dates, and identifiers that populate our catalogue are factual data, not copyrightable expression (Feist Publications, Inc. v. Rural Telephone Service Co.). Moreover, our index reproduces no textual content from the underlying books, a posture considerably more restrained than the snippet display upheld as fair use in Authors Guild v. Google (2015), where the court permitted the copying of entire books for a transformative search index.

Goodreads data was acquired by scraping publicly accessible pages without circumventing any authentication mechanism or technical protection measure. Following hiQ Labs v. LinkedIn (2022), such access to publicly available data does not constitute “unauthorised access” in the sense of the Computer Fraud and Abuse Act (CFAA). From a European standpoint, where the data was acquired, this falls squarely within the Text and Data Mining (TDM) exception for research purposes, which provides a mandatory research exception (European Parliament and Council of the European Union) that cannot be overridden by contract (Art. 7). Even if Goodreads’s terms of service were to prohibit automated access to their platform, this regulation grants academics acting in good faith and with no commercial purpose the right to do so.

Lastly, harvesting and computing data from shadow libraries as we did can be considered through the lens of several legal frameworks: In the EU, the above-mentioned “TDM exception,” and in the U.S., the fair use doctrine incorporated into the Copyright Act of 1976 complemented by the recent Text and Data Mining exemption to the Digital Millennium Copyright Act (1998). The precise application of these frameworks is a complex and evolving debate, one that is beyond the scope of this paper and our expertise as nonlegal scholars. However, from our reading, the core of the discussion, both legal and ethical, appears to hinge on the intent and context of the use.

Our work is situated in a public research institution, with no commercial interest, and aims to foster public knowledge about language, literature, and cultural representations. Applying the factors commonly weighed by these frameworks to our endeavour: The purpose is noncommercial academic research conducted in good faith; the data was accessed through publicly available channels, though the legal status of the content hosted on shadow libraries remains contested; the amount of information extracted from each book is minimal (metadata, hash signatures), serving a purely bibliographic purpose; and the resulting catalogue neither substitutes for any book we may have accessed nor affects their potential market.

Data Availability

The datasets generated and/or analysed during the present study are available in the Zenodo public data repository.⁵

Acknowledgements

The authors thank Céline Castets-Renard and Benoît de Courson for their valuable input on this research. This work was funded in part thanks to the support of PRAIRIE-PSAI (Paris Artificial intelligence Research institute—Paris School of Artificial Intelligence, reference ANR-22-CMAS-0007).

Competing Interests

The authors have no competing interests to declare.