Prof Wadan Narsey: Elections Issues 14 - Making sense of opinion polls

Professor Wadan Narsey 

The Fiji Times, 6 June 2014

Voters are being presented with the results of opinion polls by different groups of people.

Allegations are being made about polls being deliberately “biased”.

But there is no need to allege that results are being deliberately manipulated one way or another.

The reality of sample surveys is that errors in methodology can easily give “wrong” results, even if the pollsters are genuine in their intentions.

Readers will find it easier if they read through my tabular comparison of the Razor and Tebutt polls, with a genuinely good sample survey (even as low as 2%), run by the Fiji Bureau of Statistics.

Basically, if today we wanted to waste tax-payers’ money by asking all of Fiji 550,000 voters in an “Opinion Poll” to give their answers in the same secret way they will do in the polling booth in September (i.e. without telling any official how exactly they voted), of course, you will get a perfectly accurate answer, the same as you would get in the September elections.

To save some money, you could ask a genuine random sample of 20,000 voters, and you would still get a pretty accurate result of each Party’s support.

To save even more money, you could ask an even smaller 5,000 voters randomly selected by the Fiji Bureau of Statistics, to vote in the same secret way, and in my opinion, you will still get a pretty accurate answer, even if there will be some small errors.

BUT, a genuine random survey of even 5000 voters all over Fiji, urban and rural areas, including the outer islands, will cost you heaps of money, which no polling company wants to spend.

The possibility of sampling error becomes larger, as the sample size becomes smaller.

And if, to save money by not going all over Fiji, the persons polled are not truly randomly representative of all voters, then the results can be quite biased, or even worthless.

What happens then when you ask only 600 voters (Razor Group) or 1032 voters (Tebbutt Poll), using their particular methods?

What is a good opinion poll?

For any opinion poll, the possibility of systematic errors and biases depend on the following:

(1) who owns and/or controls the opinion poll? Could it lead to bias?

(2) how are the question asked and responses recorded?

(3) how randomly are the respondents selected?

(4) how many respondents are selected relative to the population of voters (which will be around 550,000)?

(5) how close might be the true party support results in the September elections, for both large parties and small?

Using these five criteria, I present a comparison of the Razor and Tebbutt polls with the independent accurate sample surveys done by the Fiji Bureau of Statistics.

Although FBS surveys are household incomes and expenditures or employment, the principles are exactly the same as in Opinion Polls.

Fiji Bureau of Statistics Sample Surveys	Razor Group		Tebbutt Poll
1. Could poll ownership bias results? The FBS is part of government, representing all Fiji, and hence usually not seen to be biased in any way. But there was a perception of bias in one Bureau survey with disastrous results. The results of the 1991 Household Survey on incomes and expenditure, so soon after the 1987 coup, was seen internally by FBS to be totally unreliable, because most responding Indo-Fijian households did not trust the Bureau interviewers (who were mostly indigenous Fijian). The Bureau never published a Report on the 1991 Survey.	The Razor Research team and the Fiji Sun are both owned by CJ Patel. The Fiji Sun receives more than a million dollars in advertising revenue from the Bainimarama Government, denied to its main competitor, The Fiji Times. CJ Patel received preferential treatment in its purchase of the Rewa Dairy Company and continues to do so, with its imports of dairy products. CJ Patel’s Financial Controller chairs some of the most powerful government controlled boards such as Fiji National Provident Fund, Fiji Revenue and Customs Authority, and the several telecommunication companies owned by FNPF. Voters naturally feel some doubt about the independence of the Razor Research results, even if they do not deliberately try to manipulate the results.		The Tebutt Polls is an independent company polling private company, but financed and the results published by The Fiji Times (as they used to do prior to the 2006 coup). A Fiji Times article recently (24 May 2014) claimed “The Tebbutt Times Poll was a scientific and objective measure of public opinion in Fiji, based on true random sampling and using globally-accepted measures and procedures.” There are some doubts about the methodology, which I suggest below in Question 3. Several prominent Bainimarama Government supporters have also claimed that the Fiji Times some years ago was biased against the Fiji Labor Party Government in 1999, and against the Bainimarama Regime from 2006.
2. What questions are asked and how? The FBS asks hundreds of questions on all major items of income and expenditure. All questions are on the Bureau questionnaire and the answers are all faithfully recorded by civil servants. Most questions are not of a sensitive nature and so householders answer quite honestly. But often some of the households, especially the rich ones, will not give the true answers on how much income they earn, or how much they spend on alcohol, or luxury goods, or other sensitive matters. So the results can be biased downwards for the rich households. But results are fairly accurate for the rest of the 95% of all households.	For both Razor Research and Tebutt Poll Respondents are asked “face to face” these questions, (1) Who is your preferred Prime Minister? (2) Which is your preferred party? Some respondents are not likely to give an honest answer to either the Razor Research Group, or the Tebbutt interviewers. As the moderate Leader of NFP has said, there is a climate of fear in Fiji where Regime critics have been punished, civil servants’ employment terminated, resources denied people who do not support government, and recently, even a scholarships terminated for the exercise of a basic human right. What respondents say to interviewers or even if they agree to sign up for the political parties registration, does not indicate how they will actually vote. The “liu muri” or “aage piche” factor is very much alive and well in Fiji amongst all ethnic groups, and also amongst all political parties.
3. How are the respondents selected? The sample households are selected in a very random and technically sound way from all the households in the country, based on the last census information. The sample comes proportionately from urban and rural and remote areas, including outer islands and Rotima, and the four divisions. Hence a truly random survey is logistically extremely difficult with Bureau staff wading through rivers, walking long distances where there are no roads, and hence very expensive (over $2 million). BUT, the results are pretty accurate about the whole country and accurate generalizations can be made about many variables. Note however, that many rich households will refuse to answer questionnaires from the Bureau, and often they are replaced by poorer households, so the results are not representative of the rich.	The Razor group is asking people around bus stations in major towns. How they are selected is anybody’s guess. The Razor Group are hoping that they will get a mixture of urban and rural people. Will that urban/rural mixture be around 50% as it is currently in Fiji? Quite unlikely and the public are not given the detailed break-downs, so we are no wiser. Will the Razor Group get all classes of voters in Fiji by asking bus travellers? Not likely as bus travellers are generally the poorer people?		The Tebbutt Poll, asks respondents from Suva, Lami, Nasinu, Nausori, Nadi, Lautoka and Ba. i.e. mostly urban Viti Levu. It is not clear how random these selections are. They may get a good break-down of responses of urban groups by ethnicity, age and gender, but not of rural Fiji or Vanua Levu. The recent Tebbutt Poll was done between Monday and Wednesday when rural people are unlikely to be in towns. If rural Fijians for example have different views on FFP/Bainimarama and SODELPA/Temumu, then even the Tebbutt Poll results will be biased probably in favor of FFP/Bainimarama. What might be the extent of the bias because of the lack of proper random sampling? We don’t know.
4. How many households/persons in the sample survey? The Bureau’s random sample numbers have been around 3,000 to 5,000 households, or about 2% to 3% of all the households in the country. A 2% sample of all 550,000 voters would require a poll of 11,000 voters randomly selected from ALL OVER FIJI. But of course, the FBS surveys ask questions about hundreds of variables, and tries to get solid results for divisions, ethnicity, urban/rural and provinces.	Since Opinion Polls ask very simple questions (as in 2 above), the sample size for the “Political Opinion Poll” can be much smaller than 11,000. But how much smaller? The Razor Team only asked 600 persons (300 were from the Central Division, 200 from the West and 100 from the North). This is probably far too small a number, and we don’t have any idea of the urban/rural, ethnicity, gender, age breakdown.		Since Opinion Polls ask very simple questions (as in 2 above), the sample size for the “Political Opinion Poll” can be much smaller than 11,000 But how much smaller? The Tebbutt Poll interviews just around 1032 persons. The results are probably accurate on urban voter views, if respondents give honest answer (see 2 above). But will rural voters vote the same way as urban voters?
5. How close are the true expected answers? If the true answers are very close, than the sample result can give you a wrong opinion. For example, the Fiji Bureau of Statistics sample survey results for average household incomes in 2008-09 were as follows: Fijian $17,000 Indo-Fijian $15,500 Others $34,000 Since “Others” includes Europeans, Part-Europeans, Chinese and Rotumans, one can very accurately say that the Others’ average household income for ALL households in Fiji is almost certainly more than that for the two major ethnic groups, perhaps double the average for Fijians. The margin for error (or the percentage of time you would be wrong) with this particular conclusion would be very small indeed, perhaps less than 1%. What about comparing Fijians with Indo-Fijians? An ordinary member of the public might say that the Fijian average household income is definitely higher than that of Indo-Fijians by $1500 or by about 10%. But the statistically smart person would remember: “hey, did I not just say that the very rich households often refuse to be part of the survey?” And “did I not say that even when they are, they under-state their true incomes to the Bureau interviewers?” So the Indo-Fijian average income from the survey is biased downwards: in reality, the true Indo-Fijian average income in all Fiji might even be higher than the average Fijian household income. i.e. the OPPOSITE of the survey results. So it can be with political parties whose support throughout Fiji is about the same: read the opposite.		For both Razor Group and Tebutt Poll Supposed that an Opinion Poll says that the “margin of error” for their results is 20% of the percentage support the poll reports for each party(usually they all claim much lower % margins of error). Suppose the poll results are: Party A has 45% plus or minus (20% of 45%) i.e. the true support could be anywhere between 36% and 54% Party C is 15% (plus or minus (20% of 18%): i.e. the true support is between 12% and 18% You can conclude, almost certainly, that the September 2014 elections will have similar ranking results. Even if there are maximum errors, the ranking will not change: Party A will have roughly three to four times as many elected persons as Party C. But suppose that the poll says that Party B has 40% plus or minus (20% of 40%): i.e. true support could be somewhere between 32% and 48% Then while the Opinion Poll reports that Party A will have more elected parliamentarian than Party B (45% is more than 40%), if you allow for errors, the reality after the September 14 elections, based on the same poll may be as follows: Possibility 1. Party A may have the lower possibility of 36% while Party B may have the higher possibility of 48%. i.e. Party B may have more in parliament than Party A (opposite of the apparent poll result). BUT neither will have absolute majority, BOTH will need a coalition to form government. Possibility 2 Allowing for errors in the other direction, Party A may have 54% (i.e. absolute majority) and can form government on its own, without any coalition. While Party B could have a mere 32%. In summary : Where a Party’s support in the September elections is going to be close to that of another Party, then the opinion poll results today may not be good predictors of who will be the larger party. This applies equally to the small parties as well.

While the Razor Research team results are not necessarily biased because of these relationships, the intelligent voters cannot avoid a perception of the possibility of bias in favour of the Bainimarama Regime, especially when the internal Razor Research processes are not available to public scrutiny.

What of other opinion polls?

There are online opinion polls run by various blogs, where the readers can click on the possible answers and the blog site automatically adds up the numbers supporting the various options.

While the respondents are anonymous, and no one knows how many times they can “vote”, the blog-sites are already known to be either opposed or supportive of the Bainimarama Regime.

Most of respondents therefore also probably have similar opinions to that of the blog-site, so the results may be inherently biased.

Sometimes, “opinion polls of taxi drivers” are used by lazy international journalists. Landing at Nadi sloshed and jet-lagged from their flights from London or Sydney, they want to want to write a quick story on Fiji’s politics, before heading off for fun and frolic to Denerau.

But 70 percent of voters do not regularly travel by taxis, hence the taxi-driver poll is also unreliable.

Then you can have the “1 person opinion poll”.

Making the rounds on the Internet currently are stories that Nostradamus five hundred years ago predicted the victory and even the name of India’s latest Prime Minister, Narendra Modi.

So here is this “Narseydamus Opinion Poll” with 3 predictions:

1.         There will be a hung parliament (i.e. no party will win more than 25 seats) with two large parties very close in the results, so there will probably be a coalition government;

2.         There will be the rise of a third party whose support might even approach that of the two large parties, with the third smaller party being the “king-maker”.

3.         At least 1 Independent candidate will get more votes than at least 10 of the parliamentarians elected under the umbrella of the larger parties.

But I would not put any of my hard-earned personal money on the “Narseydamus Opinion Poll.