New Testament Math

All Gone Quiet

2009-07-06T09:47:00.000-07:00

I have been known to abandon blogs a month in.

In this case I'm busy, but I've been working on the code to automatically identify pericopes and correspondences in the NT. Its a pretty tough problem to tackle in odd hours here or there.

Still, the data I'm getting out is almost exactly the same as the published pericope correspondences for the synoptics, which is great. Some niggly tweaks still to go.

One in particular is tricky - where to break. I think this might be a human post-processing step. We need to put a pericope boundary sometimes between words that don't appear in either pericopes. The software can't work out if the boundary should be there, or if either or both words are just omitted words from one of the bounded pericopes. I can't see any way it could possibly do that, so I think I may have to tidy the data later.

Still the Codex Sinaiticus website might eat my time to work on this for the next few days.

Codex Sinaiticus

2009-07-06T09:34:00.000-07:00

I'm sure this will be all over the biblioblogs in a few hours, but the Codex Sinaiticus project has launched their new website with the complete facsimile leaves of the codex to browse through.

I'm having trouble with speed at the moment, I guess everyone is rushing there. But it is beautiful.

You can find it at http://www.codexsinaiticus.org/

Have fun.

If you're unaware of what it is, see the Wikipedia article for an overview.

Color Map With John

2009-06-10T17:46:00.000-07:00

This is the colormap with John as well as the three synoptics.

Because of the way color-blending works, John's colors are calculated differently.

1. John only is brown.

2. Any other color in John shows shared content with the same color in the other gospels, so Yellow appearing in John shows shared content in Matt + Mark. Note that the opposite isn't true: Yellow in Matt doesn't mean it is shared with John.

3. There is no indication at all in the first three gospels of what is shared with John. The colors in the first three are purely indicating sharing between the 3 synoptics.

Clearly John hasn't got a lot of connections with the others, except for the crucifixion narrative.

Doubting Honore

2009-06-09T17:26:00.000-07:00

The image I included a few posts ago from Honore's paper on statistics in the NT...

Image via Wikipedia

...doesn't agree with my coloring map. In particular I detect a lot less triple tradition material than he does.
I'm sure this is because I am tracking words, and he is tracking pericopes. I have the pericope data (I used it to generate my map), but it will need another opportunity to analyse it.

Coloring Exercise

2009-06-08T17:06:00.001-07:00

Phew... Its been a marathon evening trying to get the code for this working well. And there's probably lots I could say about it, but for now a pretty picture.

(Image is released under a Creative Commons Attribution License 3.0, as usual).

The image is a colored map of the three synoptic gospels. Each pixel in the map represents one greek word, and it is colored depending on whether it is unique to that gospel, or is found in other gospels. Exactly as we colored the synoptics when first learning about them.

The color scheme is: Red, Green and Blue for Matt, Mark and Luke only. Yellow (Red+Green light) is Matt+Mark, Cyan (Green+Blue) is Mark+Luke, and Magenta (Red+Blue) is Matt+Luke. Black is triple tradition.

The diagram isn't sorted in any pericope-based order. Each gospel is in its correct order. I figured that would be prettier since at this scale you don't want to know what is corresponding to what.

Three things that strike me immediately (all of which I knew, but the diagram states them very clearly):

1. Mark is short and there's not much in there that's unique.

2. Luke has the most original material (John isn't here of course, if it were it would take that prize).

3. There's a lot more yellow than purple (Matt sticks to Mark more than Luke does).

And one thing I didn't realise.

4. There's twice as much magenta as cyan (Luke is closer to Matt than to Mark).

I don't know if number 4 is just one of those things that I never twigged and is perfectly obvious to everyone else, but the whole Markan priority thing made me assume that Luke would be more similar to Mark than Matt. Anyone else surprised at that? Or is it just me?

Anyway, lots of unpacking of this data to do. I'll post briefly about the techniques used to generate it, later in the week.

Synoptic Continua

2009-06-08T05:59:00.000-07:00

Thinking about the Synoptic problem from a systems approach at the weekend, I came up with a relatively simple model:

For each pair of extant documents there is a separate synoptic problem. The overall synoptic problem is the combination of these individuals.

Each pair of documents (call them A and B) sits on a continuum.

At one extreme B copied directly from A, possibly omitting some material and adding additional content.

At the other extreme A copied directly from B, again with possible omissions and additions.

In the middle of the continuum, A and B are independent and their agreements are due to some additional source.

These three cases are black-and-white and easy to draw pictures of. Filling in the continuum are other scenarios. Maybe A and B had some common sources (they'd both heard similar sermons from visiting preachers, they'd both got access to some of the same letters), and B copied from A. Now there's a blend in the relationship between both extremes of the continuum. I've tried to show this in shades of grey in the diagram.

Assuming there are 3 pairs of synoptic gospel texts - the statistical analysis of the synoptic problem should provide a point on the continuum for each of the three pairs. And (more importantly) an estimate of the error for each. It may be that the relationship between Matthew and Luke, for example, is placed somewhere between a common source (Q hypothesis) and Luke copying Matthew (Farrer hypothesis), with error margins that are significant enough that both sides can claim victory.

Would they? Or would a died in the wool Q fan think of anything less than smack in the middle as being a contradiction? From what I've read most researchers seem to pay only lip-service to the possibility of the solution being a shade of grey.

Signal to Noise Ratios

2009-06-07T05:45:00.000-07:00

In many things in science it is more important to know what you can know, than to come up with a conclusion prematurely.

One common heuristic in operations research (and science generally) is the signal to noise ratio. Data is inherently noisy. And everyone knows that the noise can obscure the signal.

What isn't widely known, is that this isn't the whole picture. In order to detect signals in the midst of lots of noise, you have to be so sensitive to patterns that you'll start to detect patterns that aren't there. In other words: to avoid false negatives (not finding something that is there), you'll be forced to used techniques that can give false positives (finding things that aren't there).

Strikes me that something of the same occurs in several scientific subdisciplines of theology. In the search for the historical Jesus, for example, we have to be so tuned to detecting theological bias in the scant records we have, that it is easy to spot features that aren't there. In the quest to solve the synoptic problem, similarly, we find features that are probably not there, which can lead to people declaring 'solutions' without understanding how likely they are to be true.

I'm not a scholar, but I'm trying to read as much on the synoptic problem as possible. So far I've not read anything that tries to seriously understand how much we can possibly know from the source text. There are various references to "of course, we'll never really know" - but nobody trying to say how much we could know.

Seems to me that is a crucial question.

The Search for Synopsis

2009-06-04T04:18:00.000-07:00

When I was thinking about starting this project, I emailed Mark Goodacre and asked some very nooby questions.

One of which was, if there is digital data anywhere on the synopses of the first three gospels. In other words, is there data on what sections in each gospel correspond to which other sections (all 4 gospels + 1 Cor I guess would be cool too). After a little bit more digging I've found various sites with summaries of the major pericope correspondences, but no science-quality data.

Image via Wikipedia

This was a little frustrating as I've been planning to do some cool stuff with that data. But then I figured it was a bit lightweight to be looking for this data anyway. In addressing the synoptic problem, one of the problems I perceive (albeit as an outsider to its scholarly study) is that it is so thoroughly overlaid with preconceived ideas. A correspondence to one person may not be to another. So I need to go synopsis hunting myself, I think, with as unbiased a statistical tool as possible.

This, unfortunately, is a lot harder that producing pretty graphs from word frequencies that I've been doing for the last week. So that's why there's a gap. And there's likely to be a bigger gap until I can spend some time at the weekend playing with it more too.

Motives

2009-05-25T14:04:00.000-07:00

This blog is intended to hold notes and result from my dalliances with computational linguistics and the New Testament.

My aim at this point is to play around with the NT in order to practice my greek and the learn more about the text. There are a couple of areas I'm interested in:

First is learning more NT greek. I learned greek in my theology degree, and have kept up with it to a modest level. But I'm still a real amateur, so I'm interested in whether mathematical engagement with the text can help me master the language quicker or more fully.

Secondly I'm interested in the Synoptic Problem, and in particular whether there are analyses that can shed light on them. I think there are some techniques from bioinformatics that could directly illuminate these problems, but getting to the point where I can do that analysis would be tricky.

Thirdly I'm perpetually interested in visualization in lots of contexts. Visualization is time consuming, but if I get time it would be good to post some graphs, charts and other images here.

How much vocab do you need to read The Reader's Greek Bible

2009-05-25T16:05:00.000-07:00

So the first question I looked at (a little warm-up really) is: what is the frequency-ordered word-list of NT greek?

The motivation for this was partly so I could get a sense of how much I'd need to know for the different 'hint' levels at John Dyer's Readers Greek Bible. If you haven't seen the site, it allows you to get footnote hints for words that appear fewer than X times in the NT. I wanted to know roughly what my vocab size would need to be to read at each level.

The answers are:

Frequency	Number of Words
100	171
90	194
80	212
70	238
60	271
50	310
45	338
40	377
35	416
30	463
25	545
20	636
15	809
10	1126

This analysis ignores word forms (so all declensions of a noun count as the same word, for example). There are 5463 different words in the NT using this method of counting.

These numbers seemed very low to me. I think they make it seem simpler than it really is - with 200 greek words you would struggle to read even with help set at level 100, because you are likely to come across novel forms of words you know. And you'd have to know the words in their strict order of occurrence, which nobody does. I'm pretty sure my greek vocab has more than a thousand words in it, but I struggle to read comfortably below level 30.

But still it is interesting.

One of the questions this raises for me, is how much vocab you'd need to know to start reading the NT.

Most readable chapters

2009-05-30T15:32:00.000-07:00

This is starting to push the bounds of usefulness, I realise, but I applied the same analysis of the most readable verses to chapters.

By the time you get to reading chapters, you need a lot of words, and it is unlikely you are going to worry about having to look up a few in the lexicon. So I realise this is artificial, but still, here are the results. The most readable chapters in the NT are:

* John 17 (the 'High Priestly Prayer' of Jesus before his arrest).

* John 16

* Rev 15 (not surprising - a very short chapter about the seven plagues).

* Rom 14

* 1 John 1 (the start of a book - many of the simplest chapters start books, as much of the language is formulaic - 1 John 1 is also very short).

Now, the thing that is most surprising about this list is just how far John 17 is ahead of the others. It it not a particularly short chapter, but is highly repetitious, and scores only 1000. John 16, coming in second, scores 2216, and then there is another gap before we get lots of chapters around the 3000 mark.

Reading through the greek of John 17, it is remarkably simple.

I think there's probably very little information in the result (no speculating that it was written by a different hand or anything like that), but I do find it interesting. And it is useful to now have a list of chapters that are pretty easy to read and that (importantly) any words I don't already know will be as useful as possible for my future reading.

FYI, the top 20 with scores are:

Position	Difficulty Score	Chapter
1	1000	John 17
2	2216	John 16
3	2753	Rev 15
4	2893	Rom 14
5	3033	1John 1
6	3033	1John 3
7	3096	John 1
8	3191	Rev 11
9	3211	Rev 17
10	3314	Matt 20
11	3417	1John 2
12	3417	1John 4
13	3424	2Cor 5
14	3431	2Thes 2
15	3478	Rom 5
16	3563	Matt 10
17	3580	Matt 7
18	3696	Rom 6
19	3781	Matt 28
20	3966	Rev 1

Verb Table Heat Map

2009-05-28T09:34:00.000-07:00

So I started playing with Grammatical analyses, as I suggested in previous posts, and I had a go at a first visualization.

It is a simple heat-map of a verb table showing what conjugations of the verb are most common and which are rare. Interestingly a whole slew of conjugations that appear in several grammars and other verb tables I have never appear at all in the NT. The optative future is the main offender, but there are several others.

If you are primarily interested in reading the GNT, this map shows what conjugations to focus on, but there are limitations, as described on the chart.

You can download it in either International A4 or US Letter format. Both are pdf files and both are around 10K in size, so they're tiny.

A Graph

2009-05-27T14:55:00.000-07:00

In my last post I talked about the number of headwords you'd need to know to start reading the NT, if you learned the words in decreasing order of frequency. (You'd need 33 to read Matt 16:15, as it turns out).

We can continue the same process for all the verses in the bible. For each verse assigning a score based on where in the frequency word list the rarest word occurs. If we order all verses in the bible using this technique we end up with this graph.

Initially in the bottom left you need to learn quite a few words before you can read another verse of the bible (it takes 33 to get to verse one, another 2 words for the next verse, then 15 more before you can read number three). Over time though the payoff begins to show. In the middle of the graph, each new word you learn gives you the ability to read between two and three more verses. Then towards the end we end up in the territory of words that only appear once, and so learning them only helps us read one extra verse.

You can think of this as three zones of learning:

1. Early undergraduate study. Everything is tough - everything seems like a special case, and nothing joins up.

2. Late undergraduate study. Suddenly you find yourself reading bits of the NT with minimal help from a lexicon. It is a great feeling.

3. The point beyond where I am. You can read about 2/3 of the NT without reference, but the words you don't know really are a special case here.

I expected the graph to have roughly this shape (flat initially, then steepening before flattening again), but I thought that the steep slope would be even steeper. It turns out that even at best (with a vocab around 1000 words) you get less than three new verses for every word you learn. On the positive side, however, the initial flat bit where progress is slow is far smaller than I expected, at this scale it is really only the smallest of ticks. After a couple of hundred words, you are into the most productive zone.

This is deeply encouraging for students of the language. Just a couple of hundred words and you can do a lot.

Of course, as I've said several times, this applies to vocab only. At some point I'll do a similar analysis of grammatical features.

PS: In case you're wondering, the little kinks at 2655 and 3495 words are where the words start to appear twice in the NT and once in the NT respectively.

Types of Math

2009-05-27T00:44:00.000-07:00

There are a few different types of math that we could do with the NT. If things go well, I might even get round to doing them.

* Firstly I'm going to be doing mostly counting stuff: frequencies, broad patterns, distributions, and so on. These are easy, because they only involve single quantities. We can look at the distribution of a particular verb form across the NT books, for example, or find the easiest verse to read (as in the previous post).

* Secondly we could look at correlations. These are data that are derived from more than one quantity, and specifically look at how they vary in combination with one another. Strictly the distribution of verbs across NT books is a correlation measure, but here I'm particularly thinking about more text-text correlations. We could look at how pairs of unusual greek forms appear together, or determine just how similar are the same pericope in different synoptic accounts.

* Thirdly we can look at models of the texts. This involves determining which of a number of models underlying the texts is more likely. So we could build a model of the synoptics with a hidden source, (Q for example), and look at whether that model is more likely than one with no hidden source. This final step is unlike the previous two because we need to construct and justify our models before we begin. Math can't magically tell us history, but it can help us understand which of several well-defined historical possibilities might be more likely, given a set of explicit (and normally quantified) assumptions.

So I'm aiming for 3, but for a while all my results will be at level 1. Mostly because I'm just getting use to the text and starting to write the code that I'm using to do these calculations. At some point between 2 or 3 I suspect I'll also need to learn a good deal more textual criticism too, but then I'm in no rush to get anywhere with this.

Word Thirty Three

2009-05-26T14:40:00.000-07:00

Following on from my previous post, I got to thinking about a "mathematically optimal greek course". One that casts aside any concept of pedagogy and optimizes for a hypothetical learning machine.

We'd want to teach this machine its vocab in roughly decreasing order of frequency. So words that appear in the NT most often are taught first. We know from the last post that after about 200 words, it would know every word that appears 100 or more times. But what does this mean.

In particular, if we learned words this way, when would we be able to read our first NT verse? And after learning 200 words, how many verses would we be able to read without help?

We can calculate this very easily. Using our frequency ordered word list we can assign each verse in the bible a score based on the lowest item in the word list. So a verse with no rare words will have a low score, and a verse containing a very rare word will score highly.

It turns out the easiest verse to read, on this scoring system, is Matt 16:15 '"But what about you?" he asked. "Who do you say I am?"' (λέγει αὐτοῖς ὑμεῖς δὲ τίνα με λέγετε εἶναι) It scores 33. So its most uncommon word (τίνα, from τίς "who") is 33 on the frequency word list. Actually all the other words in the verse are in the top 10, even in English you can see that most of them are very common (the verb λέγω "to say" is at number 9, but is the most common verb in the NT).

After Matt 16:15 comes 1 Cor 3:23 'as we are to Christ, Christ is to God' (ὑμεῖς δὲ Χριστοῦ Χριστὸς δὲ θεοῦ) with a score of 35. Then there's a small gap before we get several with scores in the 50s and 60s (some are just short, like John 10:30, others are surprisingly complex, like 1 Cor 8:6, with 27 greek words in the verse, but a score of just 50 - coming in joint 3rd overall).

Now, as in the previous post, I've ignored morphology, which skews these results. You couldn't just teach someone 33 words and have them read Matt 16:15. To get a proper curriculum for the hypothetical greek learning machine, we'd need to include grammar, and that is the subject for another post.

A Note on Resources

2009-05-26T12:15:00.000-07:00

In case you're wondering at any point. I'm using the NA27 text, as marked up and published by James Tauber at his now defunct MorphGNT site (the source texts are still available at the site, however).