Do you see any flaws in this analysis?:

“Discredited Methodology” – Nov 13, 2024
a review of
Matthew Britt and Jaaron Wingo, _Christ Before Jesus: Evidence for the Second-Century Origins of Jesus_ (2024)

The purpose of this review is to provide a response to the stylometric analysis of chs. 7-8 of Christ Before Jesus by Matthew Britt & Jaaron Wingo.
The reason for focusing exclusively on chs. 7-8 is they form the central supporting evidence of the book’s thesis that the biblical Jesus never existed.
I will point out problems with the stylometric method employed that are substantial enough to invalidate their analysis, and therefore the primary support for their conclusion.

Britt & Wingo state their thesis very plainly on pp. 1-2,

“Evidence shows that Jesus, as presented by the Gospels and New Testament books, never existed.
Despite Christianity’s claim of Jesus living during the first three decades of the first century CE, the character was actually a second-century construction.”

“Christianity, being defined as the belief in Jesus is both the Jewish god Yahweh and a historical human who was crucified and resurrected, did not emerge until the second century.
It evolved from earlier Jewish and Greco-Roman cults, and developed the narratives for its books from previous works.
The stories of a first – century Jesus movement, his disciples, and Paul are all part of a “creation myth” from the second century when the religion began to come together.”

“While small portions of the New Testament books may have been written late in the first century, evidence shows that these writings were not explicitly Christian, and in some cases, co-opted from other religious communities the remaining majority of the New Testament books were written well into the second century, with the canonical Gospels, for example, not being written until the 140s at the earliest.”

In what follows, I begin with a very brief history of the field of stylometry, emphasizing its flaws, with references.
Stemming from that, I will offer three direct critiques of Britt & Wingo’s stylometric analysis, with a specific example.

Stylometry
In order to concisely state my critiques of Britt & Wingo’s stylometry, I need to provide a little history of the field. Stylometry is the term that has come to refer to the quantitative/statistical analysis of texts. In the literature, it is often used exclusively to identify authorship of a text, but not exclusively so. According to Holmes & Kardos, “The origins of stylometry date back to 1851, when the English logician Augustus De Morgan suggested in a letter to a friend that questions of authorship might be settled by determining if one text ‘does not deal in longer words’ than another. (‘Who Was the Author? An Introduction to Stylometry’; CHANCE 16:2 (2012) pp. 5-8)” Other reviews locate the origin differently (see Tuldava, Juhan; ‘The Development of Statistical Stylistics (A Survey)’; Journal of Quantitative Linguistics 11 (2004) 141-51), but the point is that the method is old. In the definitive Ph.D dissertation on the topic for New Testament studies, James Libby writes, “[t]he early 20th century set into motion four waves of [stylometric] research (85).” In describing the last wave, he writes,

“Even though supervised techniques had begun to emerge by the early 1990s replete with [feature set extraction] techniques in tow, all was still not well in the computational stylistics kingdom. By the middle of the 1990s the seemingly isolated problems in computational stylistics in general and authorship attribution studies in particular came to a head. It wasn’t merely that the Smith/Merriam and Elliott/Foster debates had taken their toll. Nor was it that computational stylistics lacked (extending the Wild West metaphor) new mathematical horses to ride. Nor did the field lack introspective examinations of its collective malaise. The problem was that the discipline was really not yet a discipline. Reviewers such as Holmes, Forbes, Parunak, Neumann, Rudman, O’Donnell, Delcourt, Craig, Tuldava, and Juola, in particular, began to suggest or state outright that the discipline itself was deeply flawed. Rudman, for instance, famously concluded that there was “more wrong with authorship attribution studies than what was right.” Juola simply pronounced the field “a mess.” His assessment was twofold: the state of authorship attribution was dire, and authorship attribution, in particular, would fail to pass either the Frye or Daubert tests of scientific validity in courts of law. One of two outcomes now seemed certain. Either some elixir would need to be administered to the now centenarian discipline—and quickly—or its own failure to thrive would produce a natural end to one of the longest failed experiments in literary history. We organize our assessment below into four categories: the mess in findings, the mess in measures, the mess in methods, and the mess in the lack of an overarching framework (or experimental design).
(Disentangling Authorship and Genre in the Greek New Testament: History, Method and Praxis (2015); McMaster Divinity College; p. 155-6)

The first category, “mess in findings”, is that

“contradictory findings in [stylometry] were nothing new.363 The problems were so pandemic, in fact, that a frustrated Rudman collected his now famous list of eleven problems in the discipline.364 In our own discipline of Pauline studies we chronicled (per Section 2.3.3) contradictions in which Morton asserted that Paul wrote only four epistles (the Hauptbriefe), Grayston and Herdan concluded ten (all epistles but the Pastorals),365 Gerard Ledger assigned six to Paul, Barr concluded six or seven, and Kenny concluded … twelve…. (p. 157)”

The other “messes” were that bad. As a result, Libby and some other modern practitioners distance themselves from the term ‘stylometry’ and use the term ‘computational stylistics’, with an attempt to correct the errors of stylometry and use a more theoretically grounded scientific methodology designed to avoid the “messes”.

Critique of Britt and Wingo’s Stylometric Analysis

With that history in mind, we now turn to three specific critiques:

Invalid method. The stylometry of Britt & Wingo is part of what Libby referred to as the fourth wave stylometry, which was invalid. Specifically, fourth wave stylometry findings contradicted each other since they depended upon which measures and methods were selected by the authors because they weren’t theoretically grounded. Unfortunately for Britt & Wingo, this is exactly what they do in their book. For example, “There are a number of different formulas you can use, and some work better in certain scenarios than others. We found that Eder’s Delta and Eder’s Simple Delta tended to provide us with the most reliable results when looking at Greek texts of various lengths from antiquity (p. 210).” In addition to the problem of being a theoretically ungrounded methodology, their method used is single-variable, which is also shown to not be reliable in the literature. Furthermore, another problem is that their method is linguistically uninformed (see Libby, cited above, p. 157-8).

Measurements not shown. The dendrograms shown by Britt & Wingo (map of the texts that cluster the closest together – the singular statistical method used) do not include the distance measure. A striking omission since it is shown by the software’s default at the bottom of the dendrograms. This is the one statistic used for calculating their dendrograms, yet they are not showing it. Why? This removes the ability of the reader to judge for themselves how to properly interpret the results.

Genre effects are stronger than authorship. According to the literature, the stylistic effects of genre are usually the strongest – stronger than both authorship and chronology. Furthermore, genre and author effects are correlated. However, Britt & Wingo assumed that any differences they detected were due to either authorship or chronology – without ruling out the other factor, let alone genre. This is a common error made by stylometrists. Libby concludes “Specifically – and this is the major finding of this study overall – to the extent that prior work either (1) failed to test for genre as a competing theory or (2) failed to remove the effects of genre as a covariate, those studies have almost certainly confounded genre with authorship (cited above, p. 328).”

As an illustration of these flaws, consider “Exhibit 6” from Britt & Wingo p. 270. It is a dendrogram with James, Mark 13, Revelation 1, Hebrews, and Revelation 2-3, analyzed together with pieces from 1 Clement, Polycarp, Aristedes, Barnabas, Tatian, and Justin Martyr.

Stepping through the the three points of critique:

1. Method: Britt & Wingo use a single statistic to cluster a set of texts, which has been shown to generate contradictory results, depending on which statistic is used. Which Greek words were used in their analysis? There is no evidence of comprehension of the appropriate features of Greek used, let alone an effort to do so. Why was this particular set of texts selected? How many different analyses did Britt & Wingo perform in order to select the seven results shown in chapter 8 of their book? They don’t say, and this is a symptom of the fact that there was no theoretically grounded or scientifically justified design behind their search.

2. Measurement: Notice that the distance measure between the texts is not shown. Are they all relatively close to one another? Or far apart? Or does it depend on where they’re at in the dendrogram? We can’t tell.

3. Genre: This is the dominant issue. Since there are only a few genre categories of these texts (epistle, gospel, rhetoric), what is really relevant are subgenre categories, which gets into specific literary features. An inspection of the literary features explains why they cluster as they do. For our purposes, I’ll take just the five biblical texts and their closest non-biblical text in the dendrogram and point out some commonalities they share with one another, as distinct from the rest:

a. James / 1 Clement 55-59: There is a similar epistolary exhortation in both. Some shared linguistic features include addressing the audience as brothers (Gk. adelphoi, Js 1:2; 2:14; 1 Clem 57:1; 58:1), emphasis on works (Gk. erga, Js 2:14, 17; 1 Clem 57:1), exhorting patience (Gk. hupomone, Js 1:3-4; 1 Clem 58:1), and exhortation to be perfect and complete (Gk. teleioi kai olokleroi, Js 1:4; 1 Clem 56:1).
b. Mark 13 / Barnabas 16-21: Both have one person speaking regarding a temple and its destruction, with warnings.
c. Revelation 1 / 1 Clement 60-65: Both have apocalyptic language, eschatological expectation, blessings, divine revelation, and literary foci on the truth and testimony of God’s word.

d. Hebrews / Justin’s First Apology: These are longer texts. H. B. Swete notes in his ‘The Old Testament in Greek according to the Septuagint’ that both Justin Martyr and Hebrews heavily rely on the Septuagint (the Greek translation of the Old Testament), which influences their linguistic style. Craig R. Koester’s commentary on Hebrews in the Anchor Bible series highlights the theological and linguistic connections between it and Justin’s Apologies.
e. Revelation 2-3 / Barnabas 1-3: Both include a communal address, call for vigilance and faithfulness, linguistically emphasize works and deeds, and convey expectation of salvation.

You could browse the above texts in English translation, yourself, at the earlychristianwritings website. While much more could be said about the dendrogram, the above observations are derived from the texts themselves. The point is that we have shown that the text fragments very plausibly cluster due to their literary features, as opposed to authorship or chronological features. Think about it – if we added Matthew 24 and Luke 21 as separate texts, they would have clustered near Mark 13 – because of their similar literary features, not because the same person wrote them or they were written at the same time. By contrast, much of Britt & Wingo’s discussion is filled with remarks about the conclusions drawn from the dendrogram that assume the clusters imply common authorship or date.

Britt & Wingo challenge readers to prove them wrong (p. 209), but they do not provide the texts that they used, nor even state which versions they used. Why? Furthermore, they did not provide their code. Why? They could easily have hosted it on their book’s webpage. Modern researchers who seek to convince the public of novel work make it available, as is required by many scientific journals. Although they claim to be using a scientific method method (p. 208-9), they did not use it properly.

While the Britt & Wingo’s conclusions were shocking, I tried to keep an open mind as I investigated stylometry with the intention of replicating their work in order to find out whether they were right or not. It turned out that the discipline of stylometry itself is littered with contradictory results that resemble the premises of the investigator more than the hallmarks of scientific method converging upon the truth. The stylometric support cited for Britt & Wingo’s thesis is entirely invalid, as argued above. Therefore, their thesis that Jesus did not exist is without basis. Thus, I conclude my investigation here. If, dear reader, you are still not convinced, then I urge you to take a look at Libby’s dissertation (easy to find on web), particularly chapter 3.
Shalom.