Scientific Operationalisation Threatens Validity?

Black Swan · 22-06-2018 12:51am #1

Rereading One-Dimensional Man (1964) by Herbert Marcuse, which is dated in terms of some of the history and data used for arguments, but still thought provoking. Specifically, Marcuse suggests that operationalisation from theory to concept to variable measurement in the scientific method may structure research design in such a way as to threaten validity (i.e., truth in measurement). Both the physical and behavioural sciences have adopted operationalisation to such an extent that variables that are difficult to quantify may be excluded from the measurements of concepts; or, the heavy emphasis on quantification may threaten the validity of measurement, when adding some qualitative measures may improve validity (e.g., multi-method design between quantitative and qualitative measurements, and triangulation of results).

Can all variables be measured quantitatively in both the physical and behavioural sciences, while at the same time ensuring reliability and validity of measurement?

Further, to what extent does the preference for quantitative measurements (operationalisation) in scholarly journals threaten the validity of such works in addressing research problems?

Thoughts? Comments?

Fathom · 22-06-2018 9:56pm

Soft vs hard science? Both?

Fox_In_Socks · 04-07-2018 11:48am

What do you mean by operationalism?

Fox_In_Socks · 04-07-2018 12:03pm

I’ve never fully understood psychology studies. People are brought into a room to do something by an instructor but things are set up to measure something else secretly. And conclusions are drawn from this, as the participants were ‘blind’ to the actual measurement. But it’s an articificial environment to begin with. Those that are there may display similar or different personality types. What if one is really attractive or really extroverted and dominates the group beahiour?

This probably isn’t related to your discussion, sorry

Black Swan · 05-07-2018 2:03pm

Fox_In_Socks wrote: »

What do you mean by operationalism?

Briefly, reducing all measurements to mathematical operations. Precision in measurement is key, and in the natural sciences such measurements exhibit a greater level of rigour, validity, and reliability than in the social and behavioral sciences.

Black Swan · 05-07-2018 11:23pm

Fathom wrote: »

Soft vs hard science? Both?

The "soft vs hard science" distinction has become cliche. Marcuse expresses concerns regarding the scientific method emphasis on operationalisation of theory>concepts>variables to such an extent that non-mathematical measurements may be regarded as lacking merit.

From a practical standpoint, there appears to be agreement with this emphasis on mathematical operationalisation in published peer-reviewed scholar journal studies, not only in natural disciplines, but also the social and behavioral sciences; i.e., there is a greater likelihood of publishing quantitative research as opposed qualitative research. In a "publish or perish" environment, this tends to channel faculty to go mathematical in measurements, rather than submit manuscripts with subjective, qualitative methodologies.

Marcuse expresses cautions and concerns about this mathematical emphasis in science, and what implications this may have for the domination of human behavior.

The scientific method which led to an ever-more-effective domination of nature thus came to provide the pure concepts as well as the instrumentalities for an ever-more-effective domination of man by man through the domination of nature.

If what Marcuse cautions has some value, to what extent are pre-scientific philosophies being replaced by the modern and post-modern philosophies of science?

Ref:
Marcuse, H. (1964), One-Dimentional Man. Boston: Beacon Press.

Black Swan · 05-07-2018 11:43pm

Fox_In_Socks wrote: »

This probably isn’t related to your discussion, sorry

It is related, and deserves discussion.

Fox_In_Socks wrote: »

I’ve never fully understood psychology studies. People are brought into a room to do something by an instructor but things are set up to measure something else secretly. And conclusions are drawn from this, as the participants were ‘blind’ to the actual measurement. But it’s an articificial environment to begin with. Those that are there may display similar or different personality types. What if one is really attractive or really extroverted and dominates the group beahiour?

The mathematical operationalisation emphasis on social and behavioral disciplines (i.e., to get published) has resulted all too often on poorly designed measurements given yesterday's and today's methods of quantitative measurement of human behavior (e.g., Likert scales, etc.). But as the technology of measurement turns increasingly to those successful in the natural sciences (e.g., comparatively, MRI brain scans and other forms of instruments that measure the natural activity of humans) found in the emerging cognitive sciences today, which may affect the research methods used in the social and behavioral sciences tomorrow. Will human emotions and thoughts someday be measured with natural science technology instruments with rigor, validity and reliability, and if such measurements approach establishing a cause-and-effect relationship, to what extent can Pavlov's dog be replaced with human subject behavior?

It should be noted that this discussion on my part is subject to my interpretation of Marcuse in his One-Dimensional Man, and may be subject to error. Also, this is my interpretation, and not necessarily my view of science philosophy.

Fathom · 10-07-2018 9:34pm

Now reading One-Dimensional Man.

Black Swan · 10-07-2018 10:34pm

Fathom wrote: »

Now reading One-Dimensional Man.

Recommend skipping introduction completely. I would start with the "Thinking" chapters, which address the scientific method and operationalisation.

Fathom · 11-07-2018 12:11am

Beginning with "Thought..." chapters.

Permabear · 13-07-2018 9:50am

This post has been deleted.

Black Swan · 13-07-2018 5:03pm

Permabear wrote: »

This post had been deleted.

Challenging the construction and analysis of Likert scales in the social and behavioural sciences typically does not get a warm reception from its frequent users. Essentially qualitative, subjective opinions are quantified on a numbered scale; e.g., with equal intervals between greatly disagree (1), disagree (2), uncertain (3), agree (4), greatly agree (5); or some similar variation of scaled words and numbers. These scales are then treated as interval level data and subject to parametric statistical formulas accordingly (e.g., t-tests, ANOVA, Pearson r, simple linear or multiple regression). Those at the p<.05 level are treated as statistically significant, and may be used to accept or reject the null hypotheses, or in the case of regression suggest that there is an explained or unexplained relationship between independent and dependent variables used to measure a conceptual framework or algorithm.

There is a large assumption made when going from subjective words to numbers with equal intervals on a scale. For example, how about if we attempt to measure physical beauty on a scale from 1 to 10. Mary is assigned a 5 by judges on our beauty scale, while Heather is given a 10. Does this mean that Heather is precisely twice as beautiful as Mary (2 x 5 = 10), which would be at the interval level of quantitative measurement, or just that Heather is considerably more beautiful when compared to Mary; i.e., greater than, but not necessarily twice as great in terms of beauty, which would be at the ordinal level of measurement and lacking equal intervals between numbers on a scale? In Likert scales, and in our 1-10 beauty scale, the assumption made is that subjects or judges can assign with reliability and validity a quality to a quantitative scale with equal intervals, and being equal can then be added, subtracted, multiplied, and divided (or statistically analyzed as if at the interval data level) with precision, and without question.

If you point out that such opinions are really warm and fuzzy, and lacking the precision required for an equal interval scale, odds are you will be told by their users that such scales are ubiquitous and have been accepted for decades by their respective disciplines (and published in peer-reviewed scholarly journals). And if you wish to confound their opinions and judgements further, you may quote Margaret Wolfe Hungerford (1878) that “beauty is in the eye of the beholder;” or that there are probably differences between cultures in terms of what constitutes beauty, and consequently lacking the precision required for parametric statistical analysis; i.e., parametric equal interval data is one condition that may allow the results to be generalized to a population, whereas, merely greater than or lesser than ordinal data is nonparametric and cannot.

In summary, the point made by Permabear (quoted) above seems to have merit in that there are qualities that may not lend themselves well to quantification and operationalisation, and when doing so may be problematic.

Fourier · 13-07-2018 10:50pm

Black Swan wrote: »

Challenging the construction and analysis of Likert scales in the social and behavioural sciences typically does not get a warm reception from its frequent users. Essentially qualitative, subjective opinions are quantified on a numbered scale; e.g., with equal intervals between greatly disagree (1), disagree (2), uncertain (3), agree (4), greatly agree (5); or some similar variation of scaled words and numbers. These scales are then treated as interval level data and subject to parametric statistical formulas accordingly (e.g., t-tests, ANOVA, Pearson r, simple linear or multiple regression). Those at the p<.05 level are treated as statistically significant, and may be used to accept or reject the null hypotheses, or in the case of regression suggest that there is an explained or unexplained relationship between independent and dependent variables used to measure a conceptual framework or algorithm.

I think something to note is that the actual effects of this are noticeable mathematically and ties into one of my favourite links between psychology and physics.

On a simple level, many psychology papers that claim statistical significance, when reanalysed correctly actually show no such significance. There are also higher order statistical tests, beyond those of ones you have listed, which can detect a variable is not truly quantifying anything. The problem is, such techniques are not well known outside of pure mathematics and physics and are quite abstract.

A very strong example of this is CSHS inequalities in human psychology studies. These inequalities are basically (this is a bit long winded) averages of sums of products of variables, e.g. If A,B,C,D are your variables, they would be averages of things like AB + CD - AC.
If these bounds are broken, it indicates that conventional probability and statistics is meaningless to use (e.g. t-tests).

This is present very often in human cognition where classical statistics give contradictory results or show statistical significance for bizarre claims that don't conform to non-qualitative studies.

Classical statistics assumes the preexistence of well-defined values for one's variables that one is simply "extracting" in the measurement process. However for psychology this is often false. If I ask somebody "What is your favourite 80s Sci-Fi film", it is very unlikely that there was a preexistent answer to question (i.e. value to your variable), more likely the person "creates" the answer after a short bit of thinking on the spot, i.e. they decide on the answer. It's even more the case for trivial everyday questions "What drink would you like?"

However the value only coming into existence "on the spot", with the asking of the question itself, invalidates conventional statistics, this causes violations of inequalities like the CSHS ones.

So in cases like this one needs statistics which don't assume preexistent values, which is exactly the use of quantum probability, which doesn't assume this*. Hence the recent successful use of quantum theory in studies of human cognition since the works of Aerts in 1994.

*Originating there, because most likely subatomic systems don't have "context" (or "observer", as a non-technical shorthand) independent values, being most likely "created" in response to our measurements.

Fathom · 14-07-2018 5:47pm

Black Swan wrote: »

This post had been deleted.

Quantifying beauty? Something lost?

Fourier wrote: »

Classical statistics assumes the preexistence of well-defined values for one's variables that one is simply "extracting" in the measurement process.

Purpose of literature review. Intersubjectivity between studies. Used to operationally define variables. Assumes "preexistence of well-defined values." Troublesome assumption at times.

Permabear wrote: »

This post has been deleted.

Cathy O'Neil in Weapons of Math Destruction (2016) claims new, sophisticated mathematics and algorithms used to examine human behavior may be tech elite opinions embedded in code. Any merit?

Black Swan · 15-07-2018 5:38pm

Fathom wrote: »

This post had been deleted.

Permabear wrote: »

This post has been deleted.

Big data analytics often begins at the data (observations) entry point in Wallace's Wheel of Science. Some of these inquires seek emergent patterns in the data unguided by theories, conceptual models, hypotheses, or other forms of preconceived notions. If patterns are discovered, the next stage is to make empirical generalisations about that data. Some of these patterns may appear suspect, but who or what decides the patterns that appear spurious or useful?

For example, a pattern may be discovered between prior marijuana use and later hard drug use. Another pattern may emerge between prior milk drinking and later hard drug use. Although some may consider this example a nonsensical comparison, some patterns that emerge in the data are more subtle and may result in spurious empirical generalisations, which in turn may be used to influence government polices, laws, and regulations.

Marcuse was concerned by the ethics and decisions of tech elites in advancing technological society, where highly sophisticated and operationalised scientific technology successes in the control of Nature are now, or will be used, to control humanity. Such analyses have been dominated by quantitative methods, whereas ethics were in the less desirable realm of qualitative. This quantitative analytic domination represented one condition of what was becoming the foundation for One-Dimensional Man.

Fathom · 15-07-2018 8:58pm

The "Quantified Self" of research. Consistent with Marcuse. Moves from subjective, qualitative self to quantitative self. What's lost?

...increasing formalization of methodology and focus on physicality that has produced modern data-based approaches...the algorithmic body, which utilizes data gathered about individuals to compile digital representations of them. These ‘data doubles,‘ the thesis shows, are increasingly becoming a requirement of participation in modern society

Source: Loutsenko, Alexander (2016). The Self, in Numbers: Subjectivity, the ‘Quantified Self,’ and Bodies under Control. Master's thesis, Texas A & M University.

Permabear · 16-07-2018 2:13pm

This post has been deleted.

Fourier · 16-07-2018 4:02pm

Fathom wrote: »

Purpose of literature review. Intersubjectivity between studies. Used to operationally define variables. Assumes "preexistence of well-defined values." Troublesome assumption at times.

It's an extreme example of conventional statistics failing, but it illustrates the point that there are really a hierarchy of statistics that can be used to check if the current statistical model fitted to the data is valid. However it is surprising how often papers stick to the simplest ones, e.g. T-tests and don't even perform ones covered at the end of a typical statistics course, e.g. the F-test.

This is also ignoring that a model can fit a sample of data without being remotely correct. A suitably generalised GARCH model will fit almost any kind of data well, with only very obscure test statistics revealing a problem in the model. I would suggest having a look at some of the literature on predicting human choices with MRI via neural action potentials. You'll see a large discussion of the models being claimed to predict behaviour. They are constructed purely from data fitting, not from first principles and thus there is a discussion about what is really being uncovered by just fitting the data to very simple functions.

Fourier · 16-07-2018 4:09pm

I think the major differences between using these methods in the social sciences are:

You don't have complete access to all relevant variables
Often the relationship between variables isn't worked out a priori, but purely from fitting data
Battery of statistical analysis is weaker

The combination of Points 1 & 2 is a real stickler. Think of working out a prediction of General Relativity. The theory gives an exact formula as the solution, which must be matched by the data. Statistical analysis is then just to check that the data in fact fits the solution and you know the variables you want to relate.

In the social sciences though, one is trying to pull the formula out of the data without any a priori form for it. In essence one is searching the space of formula with statistics to find the best one, a much more difficult problem. Ultimately there might not even be a predictive formula as you don't have sufficient access to the controlling variables.

Fathom · 16-07-2018 5:41pm

Permabear wrote: »

This post had been deleted.

Neo-Marcuse cautions.

Permabear wrote: »

This post had been deleted.

One highly useful epidemiology method.

Permabear wrote: »

This post had been deleted.

Marcuse is not against operationalizing variables. He is against operationalization as THE dominant research method strategically and socially. Not against its tactical use in specific cases.

Permabear wrote: »

This post had been deleted.

Marcuse's technology elite dominated society.

Permabear wrote: »

This post had been deleted.

Privacy is myth for "shrugging" One-Dimensional Man. Expanding data-driven human behavior manipulation has become an omnipresent and geometrically increasing threat per Marcuse. My read. Marcuse favors methodological balance. Opposes one method domination to the exclusion of others. Advocates Hegelian competition between methods.

Fathom · 16-07-2018 6:29pm

Fourier wrote: »

...illustrates the point that there are really a hierarchy of statistics that can be used to check if the current statistical model fitted to the data is valid. However it is surprising how often papers stick to the simplest ones, e.g. T-tests and don't even perform ones covered at the end of a typical statistics course, e.g. the F-test.

From stat 101 class: "t" for 2; "F" for few (2-group significant difference comparisons vs. more than 2 group comparisons). Over-simplification of statistical assumptions, yes. Uses metacognitive skill rhyme to aid memory.

Fourier wrote: »

You'll see a large discussion of the models being claimed to predict behaviour. They are constructed purely from data fitting, not from first principles and thus there is a discussion about what is really being uncovered by just fitting the data to very simple functions.

Self-fulfilling prophecy data fit caution?

Fathom · 16-07-2018 6:42pm

Fourier wrote: »

You don't have complete access to all relevant variables

Another variation on this theme. Wonders to what extent researchers cherry pick operational definitions from literature reviews that confirm a particular position, while ignoring a significant portion of operationalizations that may contradict that position? What would Karl Popper say regarding such a practice?

Black Swan · 17-07-2018 8:31pm

Marcuse suggested operationalised technological mastery over Nature moves society from historical 2-dimensional negative-positive thinking to One-Dimensional Man positive thinking only, where tech elites proclaim that all problems can eventually be solved after an appropriate crunch of numbers. Mostly qualitative negative thinking (e.g., critical ethical questions, etc.) that challenged quantitative-based positive thinking had no utility in our Brave New World.

Polina Aronson and Judith Duportail in "The Quantified Heart" sourced from Aeon online discussed hard data driven algorithms that form the basis of artificial emotional intelligence systems. Mood managing mobile apps are said to reverse sadness to happiness, or to provide therapy-on-the go for greater emotional troubles, and have fueled emotional capitalism where happiness is equivalent to affective profit. When not mobile, half of the conversations with smart-home devices like Alexa, Siri, or Alisa have exhibited non-utilitarian existential chats between people and bots per Amazon. Machine learning progresses quickly as humans and bots engage to were Alexa, et al, may become your new best friend.

Fathom · 19-07-2018 8:19pm

Permabear wrote: »

This post had been deleted.

From pre-scientific to classical statistics to quantum methods. Has the Holy Grail of physics and mathematics finally been found in quantum theory and methods? Or will scientists 100 years in the future cast it too into the kitchen-midden of historical artifacts?

Fourier wrote: »

If I ask somebody "What is your favourite 80s Sci-Fi film", it is very unlikely that there was a preexistent answer to question (i.e. value to your variable), more likely the person "creates" the answer after a short bit of thinking on the spot, i.e. they decide on the answer.

"Bit of thinking" or preconditioned response? B.F. Skinner in Beyond Freedom and Dignity would suggest the latter. He trained pigeons and pigs to play pingpong in the Cumulative Record. To what extent do repetitious affirmations with random reward precondition responses in humans to buy products after saturation advertising? Practically, would they continue to pay for advertising if it didn't work? How does a "bit of thinking" explain impulsive buying at market checkout?

Black Swan wrote: »

Polina Aronson and Judith Duportail in "The Quantified Heart" sourced from Aeon online discussed hard data driven algorithms that form the basis of artificial emotional intelligence systems.

Emotions collected, quantified, analyzed, explained, predicted, and manipulated with an operationalized artificial emotional intelligence algorithm? Marcuse screams!

Fourier · 19-07-2018 8:26pm

Fathom wrote: »

"Bit of thinking" or preconditioned response? B.F. Skinner in Beyond Freedom and Dignity would suggest the latter.

Well actually the breaking of typical statistical bounds strongly suggests the former, which, at least in my view, is stronger than any claim Skinner might make.

Fathom · 22-07-2018 12:38am

Fourier wrote: »

Well actually the breaking of typical statistical bounds strongly suggests the former, which, at least in my view, is stronger than any claim Skinner might make.

Skinner was behaviorism theory. Statistics pertains to analytic method. Are theory and analysis different? Please clarify.

Fourier wrote: »

The problem is, such techniques are not well known outside of pure mathematics and physics and are quite abstract.

Does the "not well known... pure mathematics and physics" support the cautions of Marcuse? Especially if they are used to analyze, explain, and manipulate the many by the few (techo-elites) who are skillful in these esoteric methods?