Feeds:
Posts
Comments

Following on from previous posts, another update on the iPredict stocks for National winning the 2014 election.

Daily average trade prices are as follows:

Average daily trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

Daily average trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

And zoomed in on the last nine months, starting just before Shearer resigned as leader:

Average daily trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

Daily average trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

After topping out at $0.78 on 28 March prices for NATIONAL have stabalised in the low 70% range.

Things have not really gotten better for Labour, with the split now fairly stable at 70/30 odds over the last month.  The book is still a little asymmetrical, stronger on the bid side than the ask side, although there is a bit more room for the price to move around than there was previously.

I still think National’s chances to win are priced a little high, but at 70% it’s close enough that I’m not going to bother throwing money at it to try and fix it.

Advertisements

As explained yesterday, one of the first things we need to do is decide on the granularity for the analysis.  The New Zealand Electoral Commission makes election results available at the polling place-level, which makes them a good place to start, but difficulties occur when polling places are discontinued or new polling places are added between elections, and also when there is significant migration, such as that which occurred in and around Christchurch after the 2011 Christchurch Earthquake.  As such we need to look at where the polling places are physically located, and try to understand what happens to voter turnout between elections at the polling place-level.  This will give us an idea of where New Zealanders actually cast their votes.

But first, an aside on geographic datums.

The surface of the earth is flat, at least to first approximation.  This means that instead of dealing with spherical coordinates, such as longitude and latitude, it is often preferable to project the surface onto a plane, and treat it as a two-dimensional cartesian space where points are described by an x-coordinate and a y-coordinate (and a “height”, if necessary).  One example of such a projection is the New Zealand Transverse Mercator (NZTM) produced by LINZ specifically to represent the New Zealand mainland accurately.  Points are given as an “easting” (the x-coordinate) and a “northing” (the y-coordinate), and are expressed in meters east and north of an arbitrary reference point to the southwest of the country.  Because the NZTM is a conformal map shapes and direction are preserved, and it is therefore easy to calculate distances and bearings between nearby points.

As it happens, Elections New Zealand have handily produced a list of all the polling places used in the 2008 and 2011 elections along with their NZTM coordinates.  When we combine this information with the 2011 election party vote results by polling place we get a good idea of where people are voting.  The NZ Herald have an excellent interactive map that shows where the polling places fall under the previous 2008 electoral boundaries and the new 2014 electoral boundaries.  But we can go further and aggregate the information to look at, for example, the centre of gravity of each electorate as defined by the average of the polling place locations weighted by votes.  Results are as follows.

Clicking on the points shows the electorate name and number, total party votes (excluding special votes), and location in latitude and longitude.

One thing that immediately stands out is how sparsely populated much of the country is.  At a fundamental level the map is really only showing population density, but even then it is surprising that if you were to draw a line from Nelson to Invercargill and carve off everything to the west then that quarter-or-so of New Zealand’s land area would contain only a single electorate seat (West Coast-Tasman).  Similarly the middle of the North Island is pretty empty; if you were to cut out an area from Whakatane to New Plymouth to Whanganui to Napier you would have over one-third of the North Island and it would contain only a single electorate seat: East Coast.  (The Taupō electorate has been dragged north by polling places in Tokoroa and Cambridge.)

Of course you may think that this is a bit gimmicky, and you’d be right.  With the exception of a bit of data-vis showing election results this kind of thing isn’t of much use.  The real reasons why we want the polling place geodata are four-fold:

  1. Boundary changes.  The New Zealand Electoral Commission has just finished the latest review of electorate boundaries.  In order to simulate election results for the 2014 or 2017 elections we must figure out where each of the polling places used in the 2011 and 2008 elections are and which new electorate they now fall in.
  2. Discontinued polling places.  At every election some former polling places are discontinued, and other new polling places are added.  If we make the reasonable assumption that most people will vote at their nearest polling place then we can somewhat predict votes and turnout at polling places for the 2014 election even if they were not used for previous elections.
  3. Regression.  If we know the physical locations of each polling place, and again make reasonable assumptions about where people vote then we can at least theoretically make comparisons with the Census meshblock-level data from Statistics NZ and try and predict people’s votes based on their age, education, income, family size and so on.  A little outside the scope of my work here, but it would be a fascinating project.
  4. Get-out-the-vote.  Certain polling places will have a higher proportion than others of swing voters.  Depending on your political leanings polling place geodata will tell you where you need to concentrate your get-out-the-vote efforts.

The next post will look at political segregation in different electorates.

Appendix

I don’t have a lot of experience handling geodata, and it took a bit of effort to get the coordinate transforms into longitude and latitude working properly so that the results would show up correctly in Google Maps.  By far the best solution I found was to use the PROJ.4 – Cartographic Projections Library.  As I’m using python I needed the pyproj wrapper for PROJ.  On Mac the easiest way to get it is to use pip as follows:

curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python
pip install pyproj

There is an excellent tutorial on using geospatial data with python from the SciPy 2013 conference (see the first video).  The secret is to instantiate PROJ classes using the European Petroleum Survey Group (EPSG) Geodetic Parameter Dataset code numbers:

import pyproj
nztm_proj = pyproj.Proj("+init=EPSG:2193")
latlong_proj = pyproj.Proj("+init=EPSG:3857")
long, lat = pyproj.transform(nztm_proj, latlong_proj, easting, northing)

keywords: python, PROJ, pyproj, NZTM, Google Maps, EPSG.

 

Granularity

One of the first things you must decide upon when working on a MC simulation is the granularity of the simulation.

In particle physics the simulation would normally be at the level of the leptons, hadrons and photons we can see in the detectors, or perhaps even at the level of quarks if you are simulating plasma or collisions at higher energies.

There is a tradeoff, though, between accuracy at small scales and other factors such as computing power, data size, and also access to the raw data you need to run the model.  It wouldn’t make sense to try to simulate the weather or a tsunami at the quark-level, for example. Instead you would carve the atmosphere or the ocean up into an appropriately sized grid with granularity anywhere from a hundred meters-or-so up to tens of kilometers.

We have similar issues when trying to simulate election results, and there are a handful of fairly obvious choices for the granularity of the simulation.

  1. Nationwide: basically take the polling averages (with or without considering any margins of error) and assume that that is how the party votes will fall.  Throw in reasonable assumptions about which party’s candidates will win in Epsom, Ohariu, and the seven Maori electorates and you’ve got your result.  This is a good solution if you just want a rough guess at which side will form the government, and it is the level of reporting you typically get from the media and the blogs whenever a new poll is released.
  2. Electorate and candidate-level: This is a little more complicated.  Ideally you would want polling for each electorate, but even without that you can do an alright job by using the relative differences in results between electorates from a previous election.  This will cause problems when electorate boundaries change, however, so while it might have worked alright for the 2011 election it is a bit of a dodgy proposition for 2014.
  3. Polling place-level: The New Zealand Electoral Commission publish polling place analysis by electorate for both the party vote and the electorate candidate vote, helpfully in CSV format as well as HTML.  As with an electorate-level or candidate-level simulation you can do an acceptable job by using the relative differences in results between polling places from a previous election.  Difficulties occur when polling places are discontinued or new polling places are added between elections, and also when there is significant migration, such as that which occurred in and around Christchurch after the 2011 Christchurch Earthquake.
  4. Voter-level: Very handy if you are a political party and you want to tailor campaign material and get-out-the-vote efforts at specific individuals.  In fact, the Obama 2012 campaign data team is well known for performing simulations and doing analysis at this level of granularity. Many New Zealand libraries have copies of the Habitation Directory Habitation Index, which is the electoral role from the most recent 2011 General Election ordered by address.  Assuming you could get your hands on a digital copy then it is at least theoretically possible to geomap individual voters and make reasonable assumptions about their education, income, where they voted and who they voted for, albeit with a lot of attenuation bias.  Unfortunately if that is all the information you have to work with then that is about where you would get stuck.  If you had access to poll results with voting preferences for each person polled then things could start to get interesting, but for obvious reasons only the aggregate polling results are made available to the public.  I wouldn’t be surprised to see the two major parties working on this level of analysis and undertaking highly targeted messaging a few election cycles down the track, but I don’t think anybody in New Zealand is there yet.  Having said that, see the photo in this tweet from @somewhereben for evidence that MPs and volunteers knocking doors are already working to get their hands on some of the voter-level information that will be needed to pull this off.

I still haven’t made a final commitment to what level granularity to work at, but in the mean time I’m playing around with the polling place results try and see if we can understand what happens to voter turnout between elections at that level.  Hopefully the turnout at each polling place will be reasonably constant over time.

Motivation update

Posting has been pretty sparse since the Election.  It takes quite a lot of work to update the polls, run the simulations, and write up the results.  Given the readership – I’m guessing there’s only a couple of dozen regular readers out there – it hasn’t really been a high priority.

I’ve been on the fence for a while over whether or not I should restart the blog, but I’ve recently started learning Python and I think I need a substantial project to work on.  The plan is to start from scratch and try and create something a bit more impressive than the efforts for the 2008 or 2011 elections.

Obviously there is a bit of time pressure.  I will need to produce something reasonably polished by a couple of months before the election at the latest, but in the mean time this blog will function as a sort of how-to for those interested in how the simulations work or for those who want to cross-check the results.

If there’s any specific features you would like to see then please get in touch.

By popular request: an update of the previous post with data from the last eight days.

Average daily trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

Average daily trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

And zoomed in on the last seven months beginning on 20 August, just before David Shearer resigned:

Average daily trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

Average daily trade price, 2014 election winner stocks on iPredict for National (blue) and Labour (red).

Things have not got better for Labour, with the split now fairly stable at 70/30 odds over the last week.  What is worse is that the book is very asymmetrical: to short National’s chances of victory down to 60% you would need to drop $1813, whereas to bid them up to 80% would take only $553.  Everybody seems particularly bullish on National’s chances.

To be honest I’m a bit surprised.  In the previous post I said that I don’t see how the stock for National can realistically go higher than 70c prior to the pre-election leaders’ debates, and I stand by that.  I’m not sure why people keep bidding the stock up past 70c, although profit taking could be part of it.  I’ve left a few out of the money asks, but am starting to get a bit over-exposed.

iPredict is running a contract on National winning the 2014 election.  It was originally launched on 26 October 2011, a month before the 2011 General Election, and has been floating around between $0.40 and $0.60 since then.  It’s only in the last month that the stock has moved significantly beyond $0.60, so it’s worth taking a quick look.  Full trade history is taken from Luke Howison’s excellent API interface for iPredict, and then tweaked with a bit of Excel.

As shown in the graph below, the increase in price since has been pretty constant since it was trading at about $0.45 in October 2013, about a month or so after David Cunliffe was elected leader of the Labour Party.  The average daily price hasn’t dropped below $0.60 since 8 February 2014.

Average daily trade price, 2014 National election victory stock on iPredict.

Average daily trade price, 2014 National election victory stock on iPredict.

In  case you are wondering if this is just a short term fluctuation the answer is probably not.  Daily volumes are shown below.

Daily volume, 2014 National election victory stock on iPredict.

Daily volume, 2014 National election victory stock on iPredict.

Even disregarding the spikes you can still see that trading has increased quite a bit since, again, about mid-October.  In that four month period a total of $19200 changed hands, or about 42% of the $46100 total that has been traded since 2011.  The stock is not just bouncing around on weak liquidity.

It should also be mentioned that bid order book is quite solid: if you wanted to force the evaluation of National’s chances of winning the election back to an even 50/50 then you would need to drop a total of $1420 on 2500 contracts at an average price of $0.57.  Then you’d have to keep it there.  It’s doable, but I don’t fancy your chances.

Intra-day volatility is shown below:

Intra-day volatility, 2014 National election victory stock on iPredict.

Intra-day volatility, 2014 National election victory stock on iPredict.

The most volatile days were 28 January 2014 (David Cunliffe’s State of the Nation speech and “baby bonus”, the new flag debate), 26 October 2011 (contract launch, so meta), 28 November 2011 (first weekday after 2011 General Election), 28 October 2011, and 11 October 2012 (opposition parties’ discussion on “manufacturing jobs crisis” and Labour allegations a video exists of John Key talking to GCSB staff in February about their involvement in the Kim Dotcom case), although most on small volumes.

Daily changes are as follows:

Daily price change, 2014 National election victory stock on iPredict.

Daily price change, 2014 National election victory stock on iPredict.

The biggest absolute daily changes (other than those mentioned above) were 29 April 2012 (general debate over whether David Shearer was the correct choice for Labour Leader (see Dim-Post) and the Banks/Dotcom scandal), 22 April 2012 (a series of bad headlines for National, none of which seemed to have any affect on the polls), 13 October 2012 (“manufacturing jobs crisis”, although interestingly also the day before the MSD/WINZ security leaks exposed by Keith Ng hit the headlines?).

The 28 January 2014 in particular was an interesting day: on the day David Cunliffe was giving his State of the Nation/Baby Bonus speech stocks surged 6.6c from 59.8c to 66.4c on near-record volumes as over $1300 changed hands.  Was this the beginning of the end for Labour’s 2014 election chances?

Finally, this is the point where I’m obliged to point out that if you really think that iPredict has a pro-National bias, and that National actually have a less than a 65% chance of winning the election then you should put your money where your mouth is and go and short some stock.  There is plenty of liquidity there for you to bail out should your views change so you won’t be locked into your position long-term; you can’t use that as an excuse either.

For what it’s worth, I personally I think that barring 4-5 good poll results for Labour the stocks are going to settle at around 65c for the next few months.  I don’t see how they can realistically go higher than 70c prior to the pre-election leaders’ debates.  Having said that, I don’t think one or two good poll results for Labour are going to be enough to move them in the opposite direction either.  Labour’s prospects are looking genuinely shabby compared to four months ago (post Cunliffe election), or even 12 months ago when Shearer was still leader.

New polls are starting to come out a bit quicker now, three from Roy Morgan Reasearch (1, 2, 3), and the first post-election One News-Colmar Brunton poll. I guess I will have to go back to semi-regular once-per-month or so updates.

The updated polling averages now have National on 48.2% +/- 1.5%, Labour on 29.3% +/- 1.3%, the Greens on 13.0% +/- 0.9% and NZF on 4.8% +/- 0.6%. Changes from the previous polling update are not statistically significant!

As usual, the graphs below summarise the polling averages for the party vote after the new poll. The horizontal axes represent the date, starting 60 days before the 2008 NZ General Election, and finishing on the present day. The solid lines with grey error bands show the moving averages of the party vote for each party, and circles show individual polls with the vertical lines representing the raw statistical errors.

Party vote support for the eight major and minor NZ political parties as determined by moving averages of political polls. Colours correspond to National (blue), Labour (red), Green Party (green), New Zealand First (black), Maori Party (pink), ACT (yellow), and United Future (purple), respectively.

Party vote support for the eight major and minor NZ political parties as determined by moving averages of political polls. Colours correspond to National (blue), Labour (red), Green Party (green), New Zealand First (black), Maori Party (pink), ACT (yellow), and United Future (purple), respectively.

Party vote support for the Green party as determined by moving averages of political polls.

Party vote support for the Green party as determined by moving averages of political polls.

Party vote support for the five minor NZ political parties as determined by moving averages of political polls. Colours correspond to New Zealand First (black), Maori Party (pink), ACT (yellow), and United Future (purple), respectively.

Party vote support for the five minor NZ political parties as determined by moving averages of political polls. Colours correspond to New Zealand First (black), Maori Party (pink), ACT (yellow), and United Future (purple), respectively.

A couple of points to note. The Greens have been holding steady at about 13% +/- 1% since a month or so before the election – about 2% higher than they actually polled. National are holding steady at about 48% +/- 1%, also slightly higher than they recorded at the last election. Labour, on the other hand, look to have had a bit of a rise, peak, and fall since the election, but it’s not necessarily statistically significant. Occam’s razor says that they are polling at 28% +/- 1%, where they have been since a couple of months or so before the election.

The other point to note is the talk on Twitter and the blogs about the credibility of the recent Morgan Poll results. National, for example, went from 48.5% two polls ago down to 44.0% (a drop of 4.5%) and then back up to 49.5% (a rise of 5.5%). On the face of it this looks a bit strange, but keep in mind that the error on National’s polling is about 2%, and relative to this the fluctuations aren’t huge. On top of that, previous post-election polls had them at 47.0%, 46.0%, 45.5%, 45.5%, all well within a reasonable margin of error. Taken alone these previous results show, if anything, too little fluctuation. But whenever you deal with political polls, or any numbers with an element of randomness, you should expect periods of little fluctuation followed by some large swings. That’s just how randomness works. Given the small sample sizes on the Morgan Polls of ~800 respondents (cf. 1000 for Colmar Brunton) they seem pretty credible.