How the Brits could have used metadata to stop Paul Revere

Before I detail the next Russell Tice NSA interview — and it’s a stunner — I want to make one point about metadata and how it can be used.

According to Kieran Healy, a sociology professor at Duke University, using just metadata the British could have identified Paul Revere as a key terrorist (sorry, patriot) and apprehended him well before his storied “ride”.

Here’s how. Note that he’s writing playfully, as though he were a contemporary adviser to the British in the 1770s. The data is real though, as are the charts produced from it, and the links.

Dr. Healy begins (bolding mine, along with some reparagraphing):

Using Metadata to Find Paul Revere
JUN 9TH, 2013

London, 1772.

I have been asked by my superiors to give a brief demonstration of the surprising effectiveness of even the simplest techniques of the new-fangled Social Networke Analysis in the pursuit of those who would seek to undermine the liberty enjoyed by His Majesty’s subjects.

This is in connection with the discussion of the role of “metadata” in certain recent events and the assurances of various respectable parties that the government was merely “sifting through this so-called metadata” and that the “information acquired does not include the content of any communications”. I will show how we can use this “metadata” to find key persons involved in terrorist groups operating within the Colonies at the present time. I shall also endeavour to show how these methods work in what might be called a relational manner.

The analysis in this report is based on information gathered by our field agent Mr David Hackett Fischer and published in an Appendix to his lengthy report to the government. As you may be aware, Mr Fischer is an expert and respected field Agent with a broad and deep knowledge of the colonies. I, on the other hand, have made my way from Ireland with just a little quantitative training—I placed several hundred rungs below the Senior Wrangler during my time at Cambridge—and I am presently employed as a junior analytical scribe at ye olde National Security Administration. Sorry, I mean the Royal Security Administration.

And I should emphasize again that I know nothing of current affairs in the colonies. However, our current Eighteenth Century beta of PRISM has been used to collect and analyze information on more than two hundred and sixty persons (of varying degrees of suspicion) belonging variously to seven different organizations in the Boston area.

Rest assured that we only collected metadata on these people, and no actual conversations were recorded or meetings transcribed. All I know is whether someone was a member of an organization or not. Surely this is but a small encroachment on the freedom of the Crown’s subjects. I have been asked, on the basis of this poor information, to present some names for our field agents in the Colonies to work with. It seems an unlikely task.

If you want to follow along yourself, there is a secret repository containing the data and the appropriate commands for your portable analytical engine.

Here is what the data look like. …

Click here to see that data. Scroll down to the first Excel-type table — names are listed in the first column, and across the top is a list of “organizations of interest” — in this case, St Andrews Lodge, Loyal Nine, North Caucus, Long Room Club, Tea Party, Boston Committee, and London Enemies (yep, “London Enemies”). Healy:

As you can see, membership is represented by a “1”. So this Samuel Adams person (whoever he is), belongs to the North Caucus, the Long Room Club, the Boston Committee, and the London Enemies List. I must say, these organizational names sound rather belligerent.

Then Healy start to perform math on this table (technically, a matrix). One piece of math is to create an “adjacency matrix,” described below:

"Listen my friends and you shall hear of the midnight ride of Paul Revere"

“Listen, my children, and you shall hear
Of the midnight ride of Paul Revere”

[W]hat can get from these meagre metadata? … I will simply start at the very beginning and follow a technique laid out in a beautiful paper by my brilliant former colleague, Mr Ron Breiger, called “The Duality of Persons and Groups.” He wrote it as a graduate student at Harvard, some thirty five years ago. (Harvard, you may recall, is what passes for a university in the Colonies. No matter.)

The paper describes what we now think of as a basic way to represent information about links between people and some other kind of thing, like attendance at various events, or membership in various groups. The foundational papers in this new science of social networke analysis, in fact, are almost all about what you can tell about people and their social lives based on metadata only, without much reference to the actual content of what they say.

Mr Breiger’s insight was that our table of 254 rows and seven columns is an adjacency matrix, and that a bit of matrix multiplication can bring out information that is in the table but perhaps hard to see. …

It’s a great read, not only for the extra “e” at the end of the odd word (“networke”), but also for the spot-on data analysis. I want to give you a taste of what matrix math looks like (in prose), so one more chunk and then the conclusion. The chunk:

Take this adjacency matrix of people and groups and transpose it—that is, flip it over on its side, so that the rows are now the columns andvice versa. Now we have two tables, or matrices, a 254×7 one showing “People by Groups” and the other a 7×254 one showing “Groups by People”.

Call the first one the adjacency matrix A and the second one its transpose, AT. Now, as you will recall there are rules for multiplying matrices together. If you multiply out A(AT), you will get a big matrix with 254 rows and 254 columns.

That is, it will be a 254×254 “Person by Person” matrix, where both the rows and columns are people (in the same order) and the cells show the number of organizations any particular pair of people both belonged to. Is that not marvelous?

Yes, folks. Math majors can do this stuff, multiplying a matrix by a matrix. And here’s what you get — click for the image of relations you get after the layered analysis. And guess what that picture has at its center — a “person of interest” named Paul Revere (click, then be sure to enlarge). Remember, as the author says:

Once again, I remind you that I know nothing of Mr Revere, or his conversations, or his habits or beliefs, his writings (if he has any) or his personal life.

Really; we believe you. We also believe that The Facebook knows you’re gay, even before you do. Really.

Folks, metadata isn’t what you think it is. It’s actually data.

And in case you think this crap catches only the guilty, there’s this:

I admit that, in addition to the possibilities for finding something interesting, there may also be the prospect of discovering suggestive but ultimately incorrect or misleading patterns.

How many people name “Chris White” or something similar are on no-fly lists? You’ll never know. Unless you get stopped. Then good luck. Consider yourself collateral damage, which is totally OK with the guardians of civilization, ’cause you know, they’re guarding all civilization. (I’ll explain that; but yes, that’s how they think of themselves, and always have.)

The whole thing is fascinating, especially to us math geeks. (Dirty little secret: the NSA is loaded with math geeks. Just sayin’.)

GP

To follow or send links: @Gaius_Publius


Gaius Publius is a professional writer living on the West Coast of the United States. Click here for more. Follow him on Twitter @Gaius_Publius and Facebook.

Share This Post

  • caphillprof

    I am beginning to miss the recent days when the NSA or the White House or the Congress would make an alleged statement of fact and then you would only wait 24-72 hours and the alleged statement of fact would be revealed to be a BIG LIE.

    They have become eerily silent.

  • http://adgitadiaries.com/ karmanot

    “the British repeatedly escalated their collective punishment alienating both sides.” It’s seldom mentioned in books of Anglo history/propaganda, that this very feature of reflexive English policy made it one of the most vicious of world empires. Certainly the Irish could attest to the systemic slaughter and genocide that led to independence. The hauteur, stupidity, and hubris of the British have ironically bred perfect grounds for revolution. Now a days they are a tattered, thread bear laughing stock, burdened by a genetically stunted royal family and great victories like the Falkland’s War. —Couldn’t happen to a nation more deserving of karma. Apparently America has failed to take notes And is going down the same path.

  • MyrddinWilt

    But he did get caught. We caught him about five miles from where I am right now in Lincoln.

    The lesson I would draw from Paul Revere’s ride is not the one about metadata and PRISM but the stupidity of the British authorities which ended up losing an entire colony over a squabble about a cargo of tea. Rather than back down and accept the loss, the British repeatedly escalated their collective punishment alienating both sides.

    Massachusetts was a slaver state at the time. It was the fear that the Mansfield declaration would be applied in the colonies and in particular Massachusetts that was the real spark of the revolution. Emancipation only came about in MA in the aftermath of the war when a court interpreted the 1780 constitution in 1783.

    Strictly speaking, MA never actually abolished slavery. It was the courts interpreting the constitution that simply refused to acknowledge that slavery could exist in the state. Once the courts made clear that they would not uphold slavers property rights in civil cases and would find them guilty of assault (Commonwealth v. Jennison) in criminal cases, the institution was dead in the state.

  • http://www.americablog.com/ Naja pallida

    Funny and scary stuff, but but seems to me the point is moot. They’ve already admitted repeatedly in their blind fumbling to justify themselves that they’re collecting a hell of a lot more than just metadata. The metadata excuse was just a distraction to fool the majority of people who are only half paying attention to the problem.

  • http://buddybest.tripod.com/index.html BuddyNovinski

    OOH! Paul Revere has a French ancestry, so he could be fomenting discontent among the colonists in revenge for our victory in the Seven Years’ War. He could be pretending to be a Huguenot as a cover for a Catholic spy! We want the colonists to pay their fair share of the debt we incurred to knock the French out of North America, and he’d be motivated to use it for the French.

  • Indigo

    Fun! Maybe that explains the current British compulsion to have cameras on every street corner. Her Majesty’s government has learned from experience that an unsnoopervised population is likely to misbehave,

  • Badgerite

    Hmm. Except that the people looking to rebel and overturn the Republic these days are more likely to be militia types who make appearances often in the reports of the Southern Poverty Law Center. Also, in terms of applying modern technology to the the ‘rebellious American colonies’ model, I think the RAF would be more of a force to be reckoned with than metadata.

  • http://www.rebeccamorn.com/mind BeccaM

    Squeeeee! Beautifully clear explanation as to why metadata matters…and why it is far more than the proverbial camel’s nose in the tent.

© 2014 AMERICAblog News. All rights reserved. · Entries RSS