Serious Trump Poll Methods Problems?

Black Swan

New Harvard CAPS/Harris poll released exclusively to The Hill claims majority in poll want Trump impeached or censured (The Hill 28 December 2018).

"The Harvard CAPS/Harris Poll online survey of 1,473 registered voters was conducted from Dec. 24-26." Anyone see a problem with the days when this poll occurred?

Black Swan

Glad this OP and poll were posted just for X-mas fun and not funded research, given the lack of replies. Happy New Year!

Tree

Best time to bury some bad research apparently good to know!

Black Swan

Tree wrote: »

Best time to bury some bad research apparently good to know!

Scholars at Harvard should know better.

Tree

I sometimes find it reassuring to know that people who should know better, don't. Really takes the imposter syndrome down a few notches

Turbulent Bill

I've a bigger issue with this:

"The Harvard CAPS/Harris Poll survey is an online sample drawn from the Harris Panel and weighted to reflect known demographics. As a representative online sample, it does not report a probability confidence interval."

Essentially, we've no accuracy estimate so our results are meaningless...

Would the dates really make a difference if the sample is representative? People's political views presumably don't change just because it's Christmas.

robp

Turbulent Bill wrote: »

I've a bigger issue with this:

"The Harvard CAPS/Harris Poll survey is an online sample drawn from the Harris Panel and weighted to reflect known demographics. As a representative online sample, it does not report a probability confidence interval."

Essentially, we've no accuracy estimate so our results are meaningless...

Would the dates really make a difference if the sample is representative? People's political views presumably don't change just because it's Christmas.

It might hugely bias towards non Christians and single people, but I guess you might already being considering that.

Defunkd

Not concerned or surprised. All the authors are seeking is a headline, backed up by a poll.
A lot of Americans supposedly want him impeached but there has been no solid reason for impeachment. Does that matter? No because he's Trump..

The level of personal abuse directed toward him and his family is like nothing i've ever witnessed before but it's acceptable...because he's Trump.

American's love their polls and statistics: faulty methodology and biased/leading questions are often excused in order for a sensational result - 203% of Americans agree 92% of the time.

Black Swan

Turbulent Bill wrote: »

"As a representative online sample, it does not report a probability confidence interval."

Essentially, we've no accuracy estimate so our results are meaningless...

A notable caution indeed.

Turbulent Bill wrote: »

Would the dates really make a difference if the sample is representative? People's political views presumably don't change just because it's Christmas.

Running a poll on Christmas Eve, Christmas Day, and the day after may or may not affect the results of a study. But how do we know? It is ignored in the study design, which is troubling. Further, I failed to see this Harvard study control for the potential affects of what is a major religious holiday, as well as a major commercial holiday in America following Christmas (the latter which may or may not affect those that may elect or be available to reply to the online survey). What we try to do in our research analytics is to increase the confidence in our results, and having potential independent or intervening variable affects without controls makes our results problematic. Surely the Harvard researchers should have seen the potential for the questioning of their results given the survey's 3 day run through this major holiday? If so, why run it on those days, and not ones without this potential for spuriousness?

Black Swan

Defunkd wrote: »

American's love their polls and statistics: faulty methodology and biased/leading questions are often excused in order for a sensational result - 203% of Americans agree 92% of the time.

I am aware that there has been polling saturation in America, but I am also aware that saturation has been occurring for other nations. For example, I am working with a doctoral student who is surveying the UAE, and their center for research has warned us that survey returns have been declining over recent years due to the increasing number of surveys that have been run by universities, businesses, and government; i.e., survey saturation. Sure enough, his autumn returns were low, which may or may not have been due to UAE survey saturation.

That is not to say that rigourous survey design, administration, and results cannot be had in Ireland, America, the UAE, or elsewhere today, saturation being only one potential problem affecting this type of data collection.

Turbulent Bill

Black Swan wrote: »

What we try to do in our research analytics is to increase the confidence in our results, and having potential independent or intervening variable affects without controls makes our results problematic. Surely the Harvard researchers should have seen the potential for the questioning of their results given the survey's 3 day run through this major holiday? If so, why run it on those days, and not ones without this potential for spuriousness?

Political polls are essentially snapshots in time, rather than absolute studies. It's implicit that they'll be influenced by multiple factors unrelated to the study questions. I think holidays would be a weak factor, but who knows...

Agree in general though about controls and blinding, often poorly implemented in studies.

Defunkd

Black Swan wrote: »

I am aware that there has been polling saturation in America, but I am also aware that saturation has been occurring for other nations. For example, I am working with a doctoral student who is surveying the UAE, and their center for research has warned us that survey returns have been declining over recent years due to the increasing number of surveys that have been run by universities, businesses, and government; i.e., survey saturation. Sure enough, his autumn returns were low, which may or may not have been due to UAE survey saturation.

That is not to say that rigourous survey design, administration, and results cannot be had in Ireland, America, the UAE, or elsewhere today, saturation being only one potential problem affecting this type of data collection.

I'd hazard a guess that the USA is the most polled country on Earth- and has been for some time - so i don't know if saturation or fatigue comes into it and to what degree.
Every serious attempt at a poll/survey will address its shortcomings, margin of error etc; but it would seem that the poll you refer to here is seeking a sensational headline rather than an in-depth, serious discussion of a topic.
Michael Moore used polls to make his points about Republicans thinking that Forrest Gump was a documentary and Al Gore used data for his own purposes too. Every group in US that has a point to make
also has studies and polls that back up their point; even though there are as many 'papers' that will contradict those findings.

Black Swan

Turbulent Bill wrote: »

Political polls are essentially snapshots in time, rather than absolute studies. It's implicit that they'll be influenced by multiple factors unrelated to the study questions. I think holidays would be a weak factor, but who knows...

Anecdotal but perhaps relevant. Currently working with a doctoral student who insisted on running his pilot study during Ramadan in a Middle Eastern country. It should be noted that religion was not a concept in his conceptual framework, nor were there any controls in the research design pertaining to how religion may or may not affect the dependent variable outcomes in an online survey that was conducted (after pilot) during Autumn 2018. Pilot studies are intended to approximate the conditions of the later main survey to suggest revisions, as well as to test analytics, and by running a pilot during a major religious period in the subject country introduced a concept that was outside of the main study. Once again, how do we know what we know? And in this particular case, to what extent did the pilot conditions differ from the later survey, which may or may not have been distorted in some unknown way? From a utility and practical standpoint, why introduce something like this that may affect or not affect the later survey, when a simple delay until this major religious period has passed would eliminate potential and unnecessary threats to validity and reliability? Furthermore, parsimony (Occam's razor) suggests that research designs should provide the simplest possible (viable) explanation for a phenomenon, and avoid potential confounding factors.

Black Swan

Defunkd wrote: »

I'd hazard a guess that the USA is the most polled country on Earth- and has been for some time - so i don't know if saturation or fatigue comes into it and to what degree.

Having not reviewed the empirical evidence, if it exists, that compares survey saturation effects between countries, and perhaps offers a meta-analysis, I am uncertain if the USA was the most polled. Perhaps it was, or perhaps not?

Defunkd wrote: »

Every serious attempt at a poll/survey will address its shortcomings, margin of error etc; but it would seem that the poll you refer to here is seeking a sensational headline rather than an in-depth, serious discussion of a topic.

Once again I am uncertain if this Harvard poll had a biased agenda or not. If it was biased, I would hope that it would be subject to what Karl Popper addressed as falsifiability, a fundamental aspect of scientific inquiry, and would be treated accordingly by peer-review before publication of results, or after by the scholarly community.

Turbulent Bill

Black Swan wrote: »

Anecdotal but perhaps relevant. Currently working with a doctoral student who insisted on running his pilot study during Ramadan in a Middle Eastern country. It should be noted that religion was not a concept in his conceptual framework, nor were there any controls in the research design pertaining to how religion may or may not affect the dependent variable outcomes in an online survey that was conducted (after pilot) during Autumn 2018. Pilot studies are intended to approximate the conditions of the later main survey to suggest revisions, as well as to test analytics, and by running a pilot during a major religious period in the subject country introduced a concept that was outside of the main study. Once again, how do we know what we know? And in this particular case, to what extent did the pilot conditions differ from the later survey, which may or may not have been distorted in some unknown way? From a utility and practical standpoint, why introduce something like this that may affect or not affect the later survey, when a simple delay until this major religious period has passed would eliminate potential and unnecessary threats to validity and reliability? Furthermore, parsimony (Occam's razor) suggests that research designs should provide the simplest possible (viable) explanation for a phenomenon, and avoid potential confounding factors.

The Trump impeachment poll differs because (presumably) it's just one of a series of opinion polls on the topic over time. You assume that Christmas could be a factor, but so too could whether the participants heard the news before voting, checked their bank balance, contacted Hispanic relatives etc. - the responses are inherently volatile. Assigning controls and weights to these is impossible, and arguably unnecessary if the polls are to show opinion trends rather than absolutes.

Your student's study probably had narrower questions and only one main study, hence would need more careful controls. Even with these in place though, I'd imagine some lack of correlation between pilot and main study would be expected?

Black Swan

Turbulent Bill wrote: »

Your student's study probably had narrower questions and only one main study, hence would need more careful controls.

Several of the population parameters were known, and will allow for sample validation techniques to test to what extent the sample statistics of completed surveys were reliable indicators of these population parameters at p<.05 level of significance.

Turbulent Bill wrote: »

Even with these in place though, I'd imagine some lack of correlation between pilot and main study would be expected?

Pilot results and dry-run of analytics suggested some changes in item construction, formatting, survey administration, and statistical formulas to analyze and report results. What was not anticipated in piloting was the substantial difference in return rates between pilot and later online survey, part of which may or may not have been due to the timing of the pilot vs the later survey, the former occurring during a major religious event. Once again, from a utility and practical standpoint, why introduce a factor that could have been avoided (e.g., religious) by delaying survey administration a few days for both this student survey and the Harvard X-mas survey?