Important Disclaimer:

I am a software engineer professionally, and a political junkie, and reformed campaign-type guy. I do not have any inside knowledge here, other than my own supposition and bias. I #FeelTheBern, but will support Hillary if/when she is nominated. This is not an account of something that I know happened, but more a devils advocate case I would encourage a candidate to explore, were I advising him or her.

My background with NGP, VAN, VoteBuilder

My first exposure to VAN (Voter Activation Network - what I will call it), was in 2006 for a congressional campaign. I did not think the system filled the campaign’s needs effectively, though it was hard to argue with the future potential of the system’s ability to collect data. I last used the system in 2012, acting as the tech lead for another campaign, briefly. My general take on the system is that it really is not very well designed or engineered, but the data it encompasses is central to Democratic wins in recent years.

VAN is fundamentally a cleaned-up copy of all the registered voters in the country, along with a massive amount of data about those voters. Things like voting participation, registration and affiliation dates, local and mailing addresses, etc. are in there (and accessible to anyone paying for access to VAN). Also in there is a wealth of information from previous campaigns and polling data: “self-identifies as strongly Joe Biden, with a second-choice of Hillary,” “supports random state senator,” “identified abortion as primary voting motivation,” etc. That stuff is amazingly useful, and, for the most part, controlled/owned by the campaign that collected the data.

There is a lot there, but, I still believe VAN to be nothing more than a warehouse of data. To really use it, you have to export data and use it with more robust statistical/data analysis tools.

Because VAN is used by a wide range of candidates, and because those candidates often need support from voters who support them, but not other Democrats, the data is put into silos that can only be accessed by the data’s owner. This has been true for a long time, and, while understandable, it is maddeningly difficult to access data from other campaigns, even if that data is being made available to you.

The Story So Far

Today we heard that Senator Bernie Sanders’ staff “broke” into, and created lists of voter files based on information owned by the Hillary Clinton campaign. From everything I have seen online, this looks pretty well documented, and I think the campaign was probably correct in firing their lead data guy. The log file released by the Clinton campaign supports this, though it is obviously a human interpretation of automated audit information from VAN. I have no reason to doubt it. The DNC, based on this information, suspended the Sanders campaign’s access to the data.

It is important to note that no one is indicating that any data was “exported” from the system by the Sanders campaign. When using VAN, you are able to perform searches against a voter file, and against the wealth of stored “meta-data” the campaign has collected. These searches result in lists of voters, which can me saved/cached for later use. These lists can also be exported as Excel file (among other formats). Because VAN does not provide advanced analytical tools, most campaigns create parallel systems for performing analysis - e.g., Obama 2012’s Narwhal. I think it is worth noting that no data was exported by the Sanders campaign, but lists were simply made for reference.

Data Access Issues

Let’s be honest. At the heart of this story is the fact that the Sanders campaign accessed data owned by their competitor. That they did this is not really surprising to me. I know a lot of campaign guys, and I think many of them will make the rationalization that the ends (getting elected and promoting their agenda) will justify the means (accessing data that has questionable - I know - origin). I don’t think we need to spend a bunch of time dwelling on the fact that this actually happened. I am reasonably sure that, had he been asked, Senator Sanders or his campaign manager would have nixed the idea. Firing the guy who did it, solves the problem, primarily by making an example of the guy, and also by creating this PR shit-storm.

More important to me, however, is the fact that this could happen at all. This single issue: the security and privacy of a campaign’s data is a fundamental selling point of the VAN system. I’m not sure about the law, but it would not surprise me if it was actually illegal to share this information across campaigns. I am almost positive that VAN exposing Federal election data to state-level campaigns is illegal in Colorado. Illegal or not, though, I don’t think you can sell VAN as a service without a guarantee of privacy, security, and continuity (backups) of the data a campaign collects. It will surprise me if a guarantee of this sort is not in Senator Sanders’ Service Level Agreement with NGP.

Software Stuff

Since it’s a fundamentally important aspect of the service, it is crucial that VAN not make the mistake they made with a “bug” that exposed the data. Were I a senior-level developer or manager at VAN, in any way responsible for this “bug,” I would expect to be fired instantly. Modern software change management systems provide extensive capacity to identify engineers who contribute bad code, and modern software development practices like “test-driven development (TDD)” and “code review” are designed to prevent exactly this kind of thing from happening.

As a professional software guy, I am furious at NGP for allowing a “bug” like this to hit production. There is no way to excuse allowing a bug that exposed a customer’s data to a competing customer. None. You cannot exist as a company if you do that. I believe that NGP/VAN knows this. Have you ever heard of them splashing around donor lists? The fact that a “bug” entered the system that allowed folks to do this does not track with me.

I’ve been furious about the practices of government contractors and beltway bandits since Obama launched a slipshod healthcare.gov site. There is no excuse for launching a website that doesn’t work. I do a lot of development work with Ruby on Rails. Using that paradigm, and many others, to develop software, there are a myriad of ways in which one can quantify the functionality of the system. I can create a mechanism by which I can know that my work has 1) solved a problem, and 2) not broken something I’ve built before. That we are discussing a “bug” that breaks a previously working system suggests to me that either the people who built that system are amateurs, or that they did it on purpose. I don’t believe that you can build a complicated system like VAN, with the business needs that NGP has, and not address this question. If NGP is honestly a victim in this case, I expect their entire technology leadership team to be looking for new jobs.

NGP’s Role

The NGP argument is that a team of engineers made a mistake, and the hackers at the Sanders campaign used this to their advantage. Then, the experts at NGP were able to identify the actions of the hackers, and respond to them. On the one hand, the engineers were incompetent, and on the other they were extremely vigilant and plugged in.

To me, this pushes credibility.

Further, there has been much made of the fact that this kind of privacy breach has happened previously. This suggests that NGP does not care about breaches like this, as, if this is the case, they have not solved the problem. Either way, this suggests to me that NGP bears a great deal of responsibility for the situation.

VAN is a foundational system for a campaign. VAN has to keep data private and permanently. A breach of either of these premises suggest that VAN is untrusted. VAN cannot have a bug that breaches either of these two premises. Reports of a “bug” in VAN suggest that NGP would like to claim that this was accidental on their part, and maliciously used on Senator Sanders’ part. That, in my humble opinion, is horse shit. If the “bug” was an accident, the entire engineering team needs to be fired for misunderstanding the objectives of the business. Most engineers are smart enough to understand fundamental business objectives, though. This scenario seems a bit hard to accept.

So far, I have not heard of a single person from NGPVAN being fired.

Conclusion

NGP screwed up, badly. The Sanders campaign may have (in my opinion, probably did) tried to take advantage of this, and got caught. The Clinton campaign or the DNC saw this, and took advantage, both by cutting Senator Sanders off from VAN, and reported it to the press. I tend to view this stuff a bit cynically, and consider this to be akin to a referee missing an important call in a football game - say giving a team from Colorado an extra fourth down against Michigan, leading to a CU National Championship.

I do think there are some unanswered questions, though. We need to see the Clinton campaign’s activity logs for the same time frame before we accept that Sanders took unfair advantage. We should also ask NGP how this happened and what they are doing about it. Is VAN a secure system? It holds huge amounts of information about every single registered voter in the country, as well as a wealth of valuable political preference information.

I would also, were I a candidate using or thinking of using VAN (i.e., a Democrat running for office), I would review my Service Level Agreement with the DNC or NGP.

Saturday update

The whole situation is over. The DNC backed down and restored Senator Sanders access. Rep. Schultz claims she would have done the same thing to Hillary Clinton’s campaign. I don’t buy that. I hope Senator Sanders continues his suit, if for no other reason that to determine what actually happened.

Now, back to the 1-day news cycle.