Wrapping our heads around these word clouds
There was much discussion yesterday about the word clouds we produced displaying the results of three open-ended questions we asked about Canada’s political parties on our December national survey.
The debate has centred on whether word clouds are a useful way to present data or whether it is too simplistic. Let’s be clear: word clouds are simplistic and we decided to use them for that very reason.
Let’s discuss our approach and choice in using this somewhat new methodology.
In early December we surveyed 1,362 randomly selected Canadians from a representative panel of over 100,000 Canadians (Vision Critical’s panel). These respondents were representative of the Canadian population 18 years of age and older by region, age, gender, and language. Our sampling strategy also ensured that the sample was balanced by age, gender, and sub-region – there was a sufficient number, randomly selected, from rural, urban, and suburban areas.
The question we asked was very straightforward: When you think about the [political party], what three things first come to mind?
The questionnaire didn’t mention the party leaders’ names, their policies, or anything else about the parties. Each party was identified on a separate question page. In fact, no question prior to these questions would have biased the responses since we asked these questions near the beginning of the survey.
Therefore, the answers that respondents provided (they wrote them in small text boxes with no prompts) were not meant to be associations with a particular leader or indication of vote preference. However, as many political scientists, media commentators, and party operatives can attest, a political party rarely has a distinct image from its leader and so many of the descriptions respondents gave of the parties could be interpreted as descriptions of the party leader – but not necessarily in every case.
This was a simple branding exercise to understand where the political parties stood top of mind in early December. The temporal effects of what is in the news can affect every poll or study, whether that study is presented via word clouds or in tables. But for us, it was clear that many of the descriptions we read were longer-term impressions and not influenced by what was happening the weekend we conducted the survey.
The Word Clouds
We choose to use word clouds for two reasons: (1) they are easy to create using online software such as wordle.net, and (2) they simplify complicated qualitative data into summary graphics. This exercise was not meant to explain anything more than what Canadians think about the three main parties when they aren’t prompted by other things. And we wanted to do it in a new way that would be easy for Canadians to understand.
We translated the French responses and entered all the text into the wordle.net application unedited keeping with our desire to remain unbiased. That meant, for example, that “Steven” ended up in the Conservative Party word cloud since some Canadians spelled the Prime Minister’s name incorrectly.
We instructed wordle.net to remove commonly occurring English words and told it to only display the top 50 or so words. The word clouds do not display everything that was said about the parties – but they do represent what was said most often.
I’m the first to admit that this was not a scientific experiment. We didn’t control for partisanship, gender, or region and there was certainly no multivariate analysis involved.
The word cloud also removes some of the context of a respondent’s statement. For example, “taxes” appears in both the Conservative and Liberal word clouds, but being prefaced by ‘raising’ or ‘lowering’ would obviously change the message. That being said, just having the unqualified word shows that the issue is salient and top of mind in an absolute sense, reinforcing that we were trying to determine what people were thinking about, not how they will be voting.
However, there are times when qualitative data such as this is extremely useful (both in market and public affairs research) and I think it received so much attention because it offered another layer of understanding about why politics in Canada is the way it is.
Are the word clouds a perfect medium to explain the results? No. There are more sophisticated ways of doing it. But those methods require a level of detail and analysis that few Canadians would be willing to read.
Could we have done some things a little different? Sure. For one, in the future it may be prudent to eliminate any duplication in the clouds. Wordle.net considers a capitalized word different than an un-capitalized one. So in the future, we should merge those words together.
Are the word clouds accurate representations of sentiment towards the political parties? Yes, I think so.
Canada is a diverse country and there are no unanimous beliefs or opinions. Some people consider Stephen Harper arrogant. Some believe the Liberal Party lacks good leadership. And some people believe the NDP’s policies are unrealistic. Many others had positive things to say about each which party makes sense since many Canadians said they would vote for all three.
If someone’s gut reaction to a political party is to write “good economic managers” or “liars” that says something about what people think of the party. Mind you, it also is a good barometer of how successful the parties have been at selling a particular message or in their positioning.
Our team at Abacus Data is trying to do things differently while still keeping true to solid research fundamentals. The word clouds represent a new, and we think a fun way, to simplify complex data and make it accessible to more people.