Measuring Canonicity: Graduate Read ing Lists in Departments of Hispanic Studies

This article discusses methods to determine canonical changes in graduate reading lists using a quantitative approach. Using diversity indexes common in ecology as well as simple data exploration and distance measure techniques to study levels of canonical dominance and gender equality, we compare over 90 samples from Hispanic Studies programs in the U.S. We use these indexes to study how some of these lists have changed over time and their success in representing the diversity of experiences and social groups in Spain and Latin America.


Introduction
With the exception of a shortlived debate about the merits of Rigoberta Menchú's testimonio when it was included in the reading list of Stanford University's re quired curriculum, and occasional mentions of Borges, Neruda, and Lorca, most of the authors taught in Spanish programs mattered little for the canon wars of the late 1980s. 1 However, the opposite was the case for departments of Hispanic studies in the U.S. Both the canon wars and the rise of cultural studies in the 1990s had a significant impact on their curriculum. 2 For example, recognized periodicals in the field of Latin American studies such as Latin American Liter ary Review and Nuevo Texto Crítico published issues about the future of literary studies in Spanish, including articles focused on how to "expand their base." 3 Canon revisions inspired the first studies about gender equality and canoniza tion in graduate reading lists employed for Masters and Ph.D. examinations in Spanish. Towards the mid1990s, Joan Brown and Crista Johnson collected sev eral of these lists and analyzed their content. 4 Brown continued expanding her analysis and in 2010 published Confronting Our Canons, a booklength study on the topic. 5 Recently, Winston R. Groman collected contemporary reading lists and compared his results with those of Brown and Johnson. 6 Groman, as we will see, essentially employed the same approach as Brown and Johnson, reaching very similar conclusions.
Brown and Johnson collected a total of 56 lists. When Brown published her 2010 book she employed the same dataset she collected in the early 1990s. Ten years passed between the initial data collection and her analysis in Con fronting Our Canons, which she presented as a snapshot of contemporary His panic studies, but she did not feel the need to gather new data. Brown's expla nation is that "we have no reason to suspect that required group reading lists are radically different today-80 percent of doctoral programs still use them, and samples available online show additions to earlier versions without sub tractions." 7 In another section of her study, she defends her use of old data again, arguing that new lists are virtually identical to those from the 1990s. In her opinion, "based on the sample graduate reading lists available online today: without exception, these lists add rather than dismantle what went before." 8 Not surprisingly, her conclusion is that reading lists have changed very little in two decades: "Components of the late twentiethcentury Spanish and Latin American literary canon in the United States were all written works, and they were all in Castilian Spanish. They were overwhelmingly maleauthored, anti authoritarian, secular, heteronormative, twentyfiveyearold or older products of Roman Catholics from preferred time periods (all but the eighteenth cen tury) and favored regions: Castile in Spain, and Mexico and the Southern Cone in Hispanic America." 9 In 2016, Winston Groman gathered new data from 50 reading lists and contrasted it to Brown and Johnson's. In his conclusions, Gro man is more optimistic about Hispanic canon selections in the future, but he seems to agree with Brown that no great change has occurred: "the 18 years since Brown and Johnson's study have shown the canon makers of Hispanic Literature in U.S. universities to be aware of the limitations and prejudices of their predecessors and of the curricular changes forced by the 'Culture Wars' of the 1980s and 1990s, even if a farreaching change has not yet been realized." 10 Brown and Johnson used a system for classifying authors' canonical levels based on how many reading lists include their names. If authors are included in 90% to 100% of the lists, they belong to the "Core Hispanic Canon." If they appear in between 76 and 89 percent of the lists, they are part of the "Nearly Core Hispanic Canon." And, finally, those authors who appear in between 50 and 75 percent of the lists form the "Marginal Hispanic Canon." 11 Based on these requirements, Brown finds no women authors in the Core canon, and only three women who are present in more than 75% of the lists. 12 Groman, employ ing the same approach, reaches a similar conclusion almost twenty years later: "Women begin to appear when the percentage thresholds for canonicity are low ered, with three female authors making the Peninsular canon at a rate between 89 and 75 percent. Of the 17 authors in this fragment of the canon, only three are women, making up less than 20 percent of this subset of the Peninsular canon, and only 10 percent of the top 30 canonical authors." 13 To determine canoni cal levels, Brown, Johnson and Groman count the number of times authors are present in the lists divided by the total number of samples. We will refer to this method of measuring as the level of authorial canonicity. While we recog nize the significance of calculating the number of occurrences of authors across reading lists, we disagree with their interpretation of the data. The low number of women who achieve a significant level of authorial canonicity drives many of their conclusions as they confuse critical recognition with canonical status. In other words, from their point of view, female authors who do not appear in a high percentage of the reading lists, have not joined the canon. In her book, Brown extends this view to include other social groups or sectors. 14 High levels of authorial canonicity, though important, are insufficient by them selves for measuring the effect of canonization on graduate reading lists. Be cause for Brown, Johnson and Groman authors must achieve a specific level of presence in the lists to be considered canonical, they do not pay attention to authors whose levels are below 50%. We will be using the term canonical gesture to refer to the act of including rare authors/works in a reading list, that is, authors with a low level of canonicity. Rare authors could be either new authors that are being introduced into the canon (emerging authors) or authors whose level of canonicity is decreasing (residual authors). Whether the recipi ent of the canonical gesture reaches (o recovers) a higher level of canonicity in the future will depend on intrinsic and extrinsic factors that affect the selection of a reading list directly associated with a professional field of studies. 15 Any notion of diversity in a reading list must take into account the canonicity levels of both dominant authors and rare authors.
Obtaining a high level of authorial canonicity is not necessarily a positive value. One of the reasons for the canon wars was the need to question the idea of a static canon that transmits a specific set of values, excluding other social sec tors' points of view. The problem is not about Jorge Luis Borges or Emilia Pardo Bazán deserving to be in over 90% of the reading lists, but about those authors presenting a view of social classes that needs to be counterbalanced with experiences from other social groups. A list composed of an equal distribution of male and female authors possessing more than an 80% level of canonicity would still be an extremely static canon, with no room for new or alternative perspectives. Although the farther a list moves from dominant authors (authors with high levels of canonicity), the more diverse it tends to be, this does not mean we are arguing for the elimination of canonical authors as a way to im prove the content of reading lists.
The purpose of the present study is to look for a method to determine how much reading lists have improved in opening the canon using a quantitative approach. In our opinion, the main problem with previous studies is their methodological approach for measuring canonicity. Canonicity refers to the level of recognition of an author (or a group of authors) within a literary field. For Brown, for ex ample, this is a positive term; high canonicity signifies that an author has been acknowledged as being "great." In this study, we employ the word to describe two different but interrelated aspects of graduate reading lists. Like Brown, we use canonicity to describe the level of acceptance of an author, but, as explained above, we are interested in identifying different levels of authorial canonicity. However, we do not treat authors and their canonical levels as isolated units but as part of a whole. Thus, we also use the term canonicity to describe the spe cific combination of authors included in a list. A list is highly canonical or has a high level of canonicity if a large number of high level canonical authors are present in it. Here canonicity acquires a negative connotation as "high canon icity" means that a list is very traditional and closed to the inclusion of new and alternative voices, or it shows no signs of improving gender equity.
Our view of canonicity stems from our notion of the canon. The canon wars consolidated a move from a traditional view of the canon composed of "univer sally acknowledged" great authors to a new idea of the canon in which repre sentation becomes as important as universal recognition or consensus. When studying the reading lists, Brown, Johnson and Groman are still working with a traditional notion of the canon, which explains why for them only those authors included in most of the lists are part of it. Our study, on the contrary, is based on a postcanon wars view of the canon, a representative literary canon. This means that we are interested in canons representing a variety of points of view, but a point of view does not always have to be represented by the same author. It also means that any author who has entered the reading list is already part of the canon, no matter their canonicity level. However, for simplicity's sake, instead of talking about authors with high or low levels of canonicity, we will continue to call "canonical authors" those who possess a high level of canonic ity and "rare authors" those who possess a low level of canonicity. Similarly, we consider a list "canonically dominated" if it possesses a large number of authors with high levels of canonicity.
We are using handcurated data on reading lists employed for graduate exams in 77 institutions (92 samples) of higher education in the U.S. with Hispanic studies programs. Three of our lists are exclusively for Latin American studies programs and do not contain a Spanish Peninsular literature section. When a program has two or more lists (because they have different lists for M.A. and Ph.D. exams, or because an old list has been updated), they are considered in dependent samples. All information was collected manually from public data sources on the World Wide Web between May 2019 and May 2020, including data available from our home institution. We have chosen not to identify in stitutions by name and assign numbers to our samples as our intention is not to classify programs as doing a "good" or "poor" job with their reading lists. In the last section of our paper, when studying the evolution of reading lists over time, we assign letters to identify programs responsible for the samples we analyze. We have only focused on studying required readings, a decision that excludes Secondary Sources/Bibliography or any reading material designated as optional in a list. Literary criticism is excluded. Nonliterary or history books that have acquired cultural importance (e.g. Rama's La ciudad letrada or Ortega y Gas set's España invertebrada) are only included when they are required readings, but not if they are only mentioned as a secondary source. We will use the term "author" throughout this paper to refer to both writers and film directors. 16

Data Exploration and Methods
Our main objective is to find ways in which we can assess diversity in a reading list. Diversity must be understood in this context as the presence of a variety of canonical and rare voices. Discovering the social groups to which these voices belong and annotating our data accordingly is beyond the scope of our project. 17 Our first method for detecting levels of canonicity consisted of making obser vations about gender equity. A gender gap already highlighted in other stud ies is easily perceivable in our data, but additional observations give a more complete picture of the situation. Female authors only compose 18% of the names in the lists (15% in Peninsular literature, 19% in Latin American liter ature). 18 List size varies greatly from program to program. Figure 1 shows a dotplot chart comparing the residuals of different samples after we have sub tracted the total average of female authors from the female authors average per list (sample average = population average + residuals). Residuals as an index should be studied in conjunction with list size. The larger the residual, the more gender equity a list shows. Negative residuals correspond to male dominated lists. We expected longer lists to have the largest residuals, but that was not always the case. At the bottom right side of Figure 1, there are lists with as few as 50 authors or less, but with residual numbers between 6 and 12. However, in many programs, faculty do not seem to be taking advantage of the opportunity a longer list affords to address this problem. At the top of Figure  1, some of the medium size and large samples are the farthest away from the mean even though, having already included most of the canonical authors, they have room to compensate for the absence of women among canonical authors. For example, our largest sample (Sample 52, with a total of 351 authors) shows a residual of 3.45. While these programs still have a long way to go to create more genderbalanced reading lists, we will employ other methods for analyzing these reading lists that will allow us to see them as opening the canon in other ways (e.g. including authors that have low levels of canonicity because they are new, or considered marginal, or are creating popular or nontraditional content).
Our second method for evaluating the lists was to consider canonical percentage in their content. For this index, we assume that a reading list in which at least half of its elements are canonical authors has less room for including rare au thors or revalued/recovered documents. We have previously defined canonical authors as those who appear in at least 50% of all the lists. With this threshold, we found a total of 83 canonical authors. To obtain the relative frequency, we divided the number of canonical authors in each sample by the list size. We followed a similar procedure for rare authors, defining "rare" as an author ap pearing in less than 5% of the lists. Figure 2 visualizes the relative frequencies partially listed in Table 1.
On the one hand, longer lists mean a lower frequency of canonical authors. It also means faculty having more freedom to try canonical gestures. On the other, canonical authors tend to dominate smaller samples and allow less flexibility for the inclusion of rare names. Some of these (see bottom side of Table 1) are very traditional, containing no rare authors and over 70% of their content is composed of canonical authors. However, there are several samples that, while being relatively small, do not follow this pattern and contain a low percentage of canonical authors (Samples 87, 43 in Table 1). These outliers suggest the possibility of creating reading lists that possess a manageable size while still S T U D I E S including rare authors. We consider a sample canonically dominated when more than half of its com ponents belong to the dominant part of the canon. In 53 of the 92 samples, canonical authors dominate the selection. Domination is stronger on the male side of the list. In 65 samples, canonical authors predominate in the selection of male authors. In some cases (and not necessarily in small samples), their presence constitutes 90% of the male authors in the list. Only in 28 samples canonical authors dominate the selection of female authors. While one could argue that the disparity is the result of more male authors possessing canonical status (68 of all canonical authors are men, 13 are women, 2 are anonymous), it should be observed that the percentage of canonical authors in relation to all Although a high percentage of canonical authors in a list is evidence that it has remained traditional, one cannot necessarily equate a higher number of rare authors with a better list. A reading list for a graduate program in Hispanic studies should be a selection of authors who, for a variety of reasons internal and external are chosen to represent a field of studies and introduce students to it. Those selections are historical, not selfevident, absolute, or neutral. Even though it is theoretically possible to create a list composed of only rare authors, the result would be contrary to the idea of representative authors. A canonical gesture acquires meaning only in relation to a list of canonical authors. Thus, we defined diversity within a reading list for a literature program as the balance between the variety of its components and the presence of both canonical and rare authors.
Seeking a method for measuring a variety of content as well as canonical/rare balance, led us to adopt a diversity index often employed in ecology to measure both species richness and evenness in geographical areas. In ecological studies, Richness refers to the number of different species coexisting in a site, without taking into account the quantity of individuals belonging to one species or the other. In other words, one individual who belongs to a species of trees has as much weight as a hundred individuals from a different species. Evenness, on the other hand, refers to the distribution of different species in a sample, their relative abundance. If two samples have the same type of species, but in one of the samples a single species constitutes sixty percent of the population, then that sample is considered uneven. In ecology, "it is conventional to equate high diversity with high evenness (equivalent to low dominance)." 19 To determine diversity levels in our lists, we chose the Shannon index, which was originally designed to measure entropy, but it has become a very popular index in various fields of study such as ecology, geography and anthropology. Its purpose is to measure the probability of predicting the type or species of an individual selected randomly from a dataset.
The formula is fairly simple: R is the number of types (or species, in ecology), pi is the proportion of individuals for each type in the sample. In adapting the idea for our reading lists and determining the level of diversity of each sample, we created three types (Canonical, Rare, Others) and employed the frequency of authors in each level as the number of individuals. The Shannon index is known for its emphasis on species richness, but by grouping authors in three categories, we were able to avoid sample size influencing our numbers. 20 We employed the "diversity" function in the Vegan R package for our calculations. 21 Table 2 shows the top results for which we also computed the Pielou's evenness index. 22 There is a strong inverse correlation between diversity and canonical dominance (r = −0.91, p < 0.05). In other words, lists with a high percentage of canonical authors and few rare authors logically show lower diversity. As expected, some of the longer lists, even if they include almost all the canonical authors, appear at the top because their size allows faculty to include many rare authors. This is not surprising as making reading lists longer has traditionally been a strategy to offset the presence of traditional canonical figures. However, there were also several other lists with smaller sample sizes that obtained high numbers of diversity. This is consistent with our belief that increasing list size is not the only way to achieve a more balanced canon.

Change over Time
For a few Hispanic studies programs, we were able to acquire an updated ver sion of their reading list and compare it to the one that preceded it. This gave us an opportunity to use our methods to determine canonicity, as described above, to study temporal changes. The thirteen samples that we studied were treated as individual samples in the previous sections. We have regrouped them and assigned them letters and the year when each one was created. Program The Jaccard coefficient is defined by the size of the intersection or elements in common between two groups divided by the size of the union.

J(A, B) = (A ∩ B) (A ∪ B)
We created a matrix assigning 0 and 1 values to the lists depending on the au thors' presence or absence in the samples as exemplified in Table 3. For exam ple, when comparing GROUP 1 and GROUP 2 one would acquire the coeffi cient dividing the number of authors in common (3) by the number of authors present in both lists (7), resulting in 0.4285. The coefficient between GROUP 2 and GROUP 3 is 2/5 or 0.4, that is, there is less similitude between them than between the first two we compared. Finally, the similitude coefficient between GROUP 1 and GROUP 3 is the lowest of all (2/7 or 0.285). As we are interested in studying the difference over time between samples, we prefer to use Jaccard distance, which gives us the difference or dissimilitude between sets and is ob tained from subtracting the Jaccard coefficient from 1, or D(A, B) = 1−J(A, B).  Figure 3 employs a dendrogram to show that, based on the Jaccard distance, most updated lists in our study are still very similar to the original version. At first sight, the elapsed time between lists does not appear to have any significant effect on how much a list evolves (with the exception of E). The Jaccard distance for samples in A (0.2521008) and B (0.3833333), even though their lists were created 8 years apart are not very different from C(0.3481481), whose list was revised after two years. For D, unfortunately, we do not know the exact dates of revisions, but the distance (0.3666667) is similar to A, B and C. Based on these numbers, one could argue that changes within a canon are never drastic, no matter how much time has elapsed between versions. The exception is E which has changed so much (0.7236842) that the updated version no longer resembles the original one. However, it appears that E has been affected not by the time that passed between list creation dates, but by size reduction (it changed from 138 authors to 56). This can be seen in Table 4, where we have contrasted all of our indexes to study the evolution of these lists. The updated list from A has slightly increased in size and its Shannon index and Evenness have improved noticeably. Another positive aspect is that it is less dominated by canonical authors. These are significant changes in spite of the small Jaccard distance between samples. However, it is concerning that the gender gap has increased (goes from being 1.72 above the average to 1.19). A closer look at the elements that have been added or deleted can explain these changes. Only 5 authors are eliminated from the 2007 list (one of them with a level of canonicity above 70%) and 24 are added in 2015. Out of those 24, there are only 4 women, and 9 of them, including one of the women, are film directors. This late effect of cultural studies on the Hispanic canon might not be an isolated phenomenon. For example, in the samples from B, also with a gap of 8 years between them (2009 and 2017), 11 authors are eliminated, including 3  (24 %), and 16 are film directors (including two women). Its three indexes improved, but the gender gap remained about the same. In both samples, the inclusion of film directors, whose authorial canonical level is not very high, had the effect of improving all the indexes, but it is obvious that adding nonliterary authors (a vast majority of which are male) took priority over including more women authors when updating these lists. Samples from D show that this type of effect can be avoided. 10 authors were eliminated with the update (7 men, 3 women) and 30 were added (a third of which were women). Even though 17 film directors (2 of them women) were added, not only indexes improved and canonical dominance decreased with these changes, but gender equality improved.
The inclusion of film and nonliterary documents is not the only reason for change. Making the lists shorter and more manageable for students is the main aspect behind the transformation of C and E. But while C improved its Shannon index and greatly increased the participation of female authors in spite of hav ing eliminated 37 authors (added 10, 5 women), the radical transformation that E underwent turned it into a more canonical list. E eliminated 96 authors (24 women) and added 14 new ones (6 women). Diversity decreased and the gen der gap became more pronounced. E's effort to provide students with a more manageable list resulted in decreasing the number of rare authors and creating a more traditional list.
Perhaps the most interesting samples for our study were those from F. This uni versity program has a list limited to 40 authors/titles which, it appears, must be updated every three years. As a result, we have three samples which cor respond to the threeyear periods of 20132015, 20162018, 20192021. All samples are supposed to be about the same size, but some of the titles are an thologies from which students are directed to read specific authors. Following our data collection method, these were added as individual authors making the lists slightly uneven. Unlike other samples, which have slowly changed for ped agogical or practical reasons, the forced changes in F are much more drastic and occur within a relatively short period. Between the first list (F 201315) and the second (F 201618) the number of authors removed or added remains the same. The same number of women authors removed are added, canonical authors are removed (e.g. Juan Rulfo) only to be replaced with others (e.g. Mario Vargas Llosa). While it seems a wellintended effort to improve the process of updating reading lists, it has the effect of maintaining the status quo and all indexes re main about the same. Making sure that any woman author eliminated is replaced does not address the gender gap. The Jaccard distance between the second (F 201618) and third (F 201921) samples is one of the highest (0.6212121, only E is higher), due to the elimination of the two anthologies containing rare au thors. 27 entries were eliminated, 11 of them rare; 11 new authors added, 5 rare. Diversity and evenness decreased as canonical dominance increased sig nificantly. While we find the model laudable for many reasons, indexes for F show that it is not enough to "keep the same numbers" between updates.

Conclusions
We believe that the use of the Shannon diversity index in conjunction with sim ple observations about levels of dominance and gender equality can be a simple but effective way to quantitatively detect the canonical level of graduate reading lists. Reading lists in departments of Hispanic studies in the U.S. have not re mained static as the earlier research in this area concluded. They are becoming less traditional and the faculty in charge of creating them are including new au thors, representing a greater diversity of experiences and social groups in order to present a more accurate picture of the societies in Spain and Latin America. Unfortunately, they are not updated regularly. Most importantly, the problem of gender parity persists and needs to be urgently addressed. This is not a problem unique to Hispanic Studies, but a systemic one in academia. Studies of other kinds of lists (syllabi, journal publications, bibliographies) in other disciplines (e.g. social sciences, International Relations, Political Science) have consis tently found similar results: about 70 to 80 percent of their lists are composed of male authors. 24 The gender gap in the graduate reading lists is the result of an "intricate web of systemic exclusion" of female authors created by the inter action of "institutional, disciplinary and (possibly) individual biases . . . and stereotypes around gender and knowledge." 25 This could be easily seen (though more data is certainly needed) in how the late influence of cultural studies on some of the lists studied in the preceding section becomes an excuse for not focusing on increasing the representation of women authors. While many His panic studies departments are moving away from the use of lists for their Ph.D. exams and experimenting with new formats, it is still the norm to use them for M.A. examinations and improving their content should be a priority for these programs. Expanding a reading list is often a preferred method of opening the canon, but it is not the correct approach. Not only is it unrealistic to ask stu dents to study between 200 and 300 authors, but despite the fact that longer lists can potentially increase diversity in points of view, they often protect rather than challenge the presence of canonical authors. Additionally, they increase the number of women authors without addressing the gender gap. While short reading lists tend to be dominated by canonical authors, it is possible to achieve a high diversity and evenness with a limited number of authors (see Table 2). Simple steps such as revising the lists regularly (every three years, for example, following the model implemented in at least one program) and verifying that there is a balanced presence of canonical and emerging voices, male and female authors, would greatly improve some of the lists being used today. The data and methods used for the present study will hopefully help faculty members in insti tutions of higher education in identifying and challenging canonical hierarchy.