Comparison of means data collection mess up

The_Scary_Man

Hi,

I am currently writing up my thesis based on an intervention with a group in which I was hoping to analyse pre-intervention and post-intervention measures.

In terms of my methodology I asked a group of 58 athletes to complete the measures. I then delivered a four part course over a period of 6 weeks and then asked the athletes to complete the measures again.

My problem is that in the pre-test measures I never got identifying information from any of the participants so I can't match up the before and after results by participant.

I've been trying to figure out what my options are online but I'm struggling to understand what is available to me.

I don't think I can use a dependent t-test as I can't match up the measures and I don't think I can use an independent t-test as the measures are related.

I would be grateful if anyone could let me know if there are any other options available to me?

Thanks,

Ken

Black Swan

Hi Ken. Welcome to the Researcher forum. Please feel free to return to ask several questions until you find solutions to your research problem.

The_Scary_Man wrote: »

I am currently writing up my thesis based on an intervention with a group in which I was hoping to analyse pre-intervention and post-intervention measures.

In terms of my methodology I asked a group of 58 athletes to complete the measures. I then delivered a four part course over a period of 6 weeks and then asked the athletes to complete the measures again.

We need more information and details regarding the conduct of your study before we can offer a more useful recommendation. Given this lack of sufficient information, I will attempt to address some of your questions, but with caution.

What experimental design did you use? Did you have randomly assigned control subjects in addition to your 58 subjects that received the treatment? If no controls, then your design is pre-scientific and this should be noted in the study limitations. This sometimes occurs today in preliminary and exploratory studies, with a recommendation in your thesis conclusions and recommendations that further study is needed with perhaps a larger sample size and more rigorous design (e.g., Solomon 4-group experimental design, or some other control design).

The_Scary_Man wrote: »

My problem is that in the pre-test measures I never got identifying information from any of the participants so I can't match up the before and after results by participant.

This is problematic. Group unit of analysis may be your approach, not individual unit of analysis.

The_Scary_Man wrote: »

I don't think I can use a dependent t-test as I can't match up the measures and I don't think I can use an independent t-test as the measures are related.

Compare group means between pre-test and post-test to see if there was a significant difference at p<.05 level of significance.

Not knowing your data levels for each test measure makes further recommendation problematic (e.g., nominal, ordinal, interval, ratio). Furthermore, the small n=58 may not allow for establishing if your variables were normally distributed during the description stage of your analysis. If not normally distributed due to the limitations of small-n, or some other reasons, it may be wise to use non-parametric statistics rather than parametric. If non-parametric, you could only discuss changes between groups, and not generalise your results beyond the 58 subjects, which should also be noted in your results, limitations, and conclusions.

Once again, caution should be used when reviewing these above recommendations given that I am lacking sufficient information.

The_Scary_Man

Hi Black Swan,

Thanks for taking the time to respond

I'm very new to the research game so please bear with me if I am a bit slow in grasping some of the concepts.

The design is pre-experimental, I had access to a specific group of athletes over a specific time-frame and due to time constraints and the requirements of the organisation involved there wasn't the option of creating a control group.

The research is concerned with investigating the impact, if any, of a cognitive behavioural group coaching intervention on intrinsic motivation in elite level 17 to 19 yr old athletes. Due to the large size of the group it was split into two groups by age to facilitate delivery of the intervention.

Interventions like these are best delivered at an individual level and I wanted to see if delivering it at a group level would have an impact as most clubs or organisations wouldn't have the resources to facilitate one to one programs for their members.

I administered two measures prior to beginning the sessions, the Athletic Coping Skills Inventory (ASCI-28) and the Sport Motivation Scale (SMS-28), both using Likert scales. The ASCI-28 gives an overall score of the athletes psychological skills with 7 subscales of specific psychological skills. The SMS-28 measures athlete motivation under 7 categories spanning intrinsic motivation, extrinsic motivation and amotivation.

I then delivered four 45 min group sessions to each group over the course of 6 weeks and met the groups one week after the final session to collect the second set of measures.

I have played around with a few options on SPSS, such as the Wilcoxon Signed Rank Test due to the small n=58, and I explored using Cohen's d to measure effect size.

The Wilcoxon Signed Ranks test is showing a difference at p<.05 in the overall ASCI-28 score and also in two of the seven sub-scales (see attached pic).

In the SMS-28 only the total Extrinsic Motivation score showed a difference at p<.05.

As I had identifying information for the second set of tests I also checked for any correlation between the ASCI scales and the SMS scales using Pearson's Correlation.

As you can see, in my panic I'm employing a bit of a 'chuck it at the wall and see what sticks' approach and I'm just not sure that any of these options are giving me any valid information.

Black Swan

The_Scary_Man wrote: »

The design is pre-experimental, I had access to a specific group of athletes over a specific time-frame and due to time constraints and the requirements of the organisation involved there wasn't the option of creating a control group.

We could attempt to increase the rigour of your data set, if you wish. I am contemplating the possibility of establishing a quasi-control group ex post facto. In addition to your treatment pre-and-post measures, did you collect any demographic data for your n=58 subjects? You mentioned age, any other demographics (e.g., year in school, ethnicity, residence, GPA, etc.)?

How long ago had this data been collected? Was it recent, or many months or year ago?

Were there similar athletes that were not included in the n=58 and did not participate in the study? If so, how many, and can you access them now? If your data collection had occurred relatively recently, would it be feasible to administer the post-test only to these athletes that did not participate with the original n=58? In terms of sample size, it would be grand if you could measure 20 or more additional quasi-control subjects, but we could work with a minimum of 10, but not less. I have a citation for a minimum of 10 subjects for preliminary hypotheses testing (Isaac, S, and Michael, W., Handbook in Research and Evaluation. San Diego: EdITS Publishers) and can find the page if you need it.

The_Scary_Man wrote: »

The research is concerned with investigating the impact, if any, of a cognitive behavioural group coaching intervention on intrinsic motivation in elite level 17 to 19 yr old athletes. Due to the large size of the group it was split into two groups by age to facilitate delivery of the intervention.

I am familiar with CBT, as well as with coaching, but not both combined. I have also been involved in research with motivation concepts and operationalised variables. Hopefully that may make me useful to you.

Another question. How precisely did you split your n=58 subjects into 2 groups by age? All 17 and 18 in one group, 19 in another, or sorted into 2 equal sized groups by DOB, or what? Also, sample size for each group? Depending upon how this was done, you may be able to control or otherwise compare for the effects of age. Please advise.

We can address the other questions you had in bite-sized pieces, rather than load all of them at once. Once again, not having free access to all your raw data, methods, etc., caution should be exercised with my recommendations.

I don't have all the answers, and other researchers reading this thread are most welcome to comment, ask questions, offer differing recommendations, or whatever. We can advance by sharing, and hopefully help our OP.

The_Scary_Man

Black Swan wrote: »

We could attempt to increase the rigour of your data set, if you wish. I am contemplating the possibility of establishing a quasi-control group ex post facto. In addition to your treatment pre-and-post measures, did you collect any demographic data for your n=58 subjects? You mentioned age, any other demographics (e.g., year in school, ethnicity, residence, GPA, etc.)?

Unfortunately, I didn't think to collect any demographic data for the group.

How long ago had this data been collected? Was it recent, or many months or year ago?

I collected the data and delivered the intervention over July and August of this year.

Were there similar athletes that were not included in the n=58 and did not participate in the study? If so, how many, and can you access them now? If your data collection had occurred relatively recently, would it be feasible to administer the post-test only to these athletes that did not participate with the original n=58? In terms of sample size, it would be grand if you could measure 20 or more additional quasi-control subjects, but we could work with a minimum of 10, but not less. I have a citation for a minimum of 10 subjects for preliminary hypotheses testing (Isaac, S, and Michael, W., Handbook in Research and Evaluation. San Diego: EdITS Publishers) and can find the page if you need it.

I can speak to the organisation to see if there are any more athletes that I can include but I don't think there will be. My own concern would be that the athletes were part of a Talent Identification and Development (TID) program and all of the athletes involved took part, this was one of the reasons that we could not split off a control group as there was a concern that participation by only part of the group could unfairly skew the talent identification process, so any additional athletes would have to come from outside the program.

This is to a degree the main stage that these athletes have to make an impression on the Academy staff and so we decided it was more ethical to offer all of the athletes the opportunity to take part in the research.

I am familiar with CBT, as well as with coaching, but not both combined. I have also been involved in research with motivation concepts and operationalised variables. Hopefully that may make me useful to you.

That is great, thank you Black Swan. Cognitive Behavioural Coaching is basically the application of CBT techniques in a non-therapeutic setting. Cognitive Behavioural interventions form a large part of psychological skills training programs.

We delivered four 45 min sessions to each of the groups;

Session 1 - Introduction to Mental Skills
Session 2 - Personal Management (Emotional Awareness, System 1 and System 2 Thinking, Thought Stopping and Reality Testing)
Session 3 - Goal Setting
Session 4 - Self Talk - Managing Your Internal Conversation

Another question. How precisely did you split your n=58 subjects into 2 groups by age? All 17 and 18 in one group, 19 in another, or sorted into 2 equal sized groups by DOB, or what? Also, sample size for each group? Depending upon how this was done, you may be able to control or otherwise compare for the effects of age. Please advise.

When I went back in to look at my data I forgot that I had to remove one dataset due to an incomplete form so my overall n=57. We didn't split the groups by age per se, the group was actually made up of two squads, the under 19s n=31 and the u20s n=26. The u20s would have had an extra year in the TID program but no exposure to psychological skills training. There were some differences in the pre-test between the groups but less so in the post-tests.

We can address the other questions you had in bite-sized pieces, rather than load all of them at once. Once again, not having free access to all your raw data, methods, etc., caution should be exercised with my recommendations.

Would it be acceptable from an ethical perspective to share the anonymised raw data?

Black Swan

The_Scary_Man wrote: »

I collected the data and delivered the intervention over July and August of this year.

This is relatively recent.

The_Scary_Man wrote: »

I can speak to the organisation to see if there are any more athletes that I can include but I don't think there will be. My own concern would be that the athletes were part of a Talent Identification and Development (TID) program and all of the athletes involved took part, this was one of the reasons that we could not split off a control group as there was a concern that participation by only part of the group could unfairly skew the talent identification process, so any additional athletes would have to come from outside the program.

Your concerns seemed justified. Adding athletes from outside the TID program does not have merit. It appears that you have a nonprobability sample of subjects contained within the TID parameter which can only be used to represent those N=58 subjects that were included in the study; i.e., you cannot generalise your results beyond those N=58 participants (or n=57 with one subject being excluded for missing data). A nonprobability sample suggests that you should use nonparametric statistics to describe and explain your results. Nonparametric statistics would also fit well with the ordinal scale level of your survey measures (i.e., you would not use t-tests, which are parametric).

I would suggest that you note in limitations that caution should be exercised when interpreting these results in terms of external validity; i.e., only representative of n=57 subjects and not generalisable to a larger population.

The_Scary_Man wrote: »

When I went back in to look at my data I forgot that I had to remove one dataset due to an incomplete form so my overall n=57.

Loss of 1-subject for missing data should be noted, but it should not affect the results substantially (N=58; n=57).

The_Scary_Man wrote: »

We didn't split the groups by age per se, the group was actually made up of two squads, the under 19s n=31 and the u20s n=26. The u20s would have had an extra year in the TID program but no exposure to psychological skills training. There were some differences in the pre-test between the groups but less so in the post-tests.

Maturation may be a factor affecting your internal validity? If so, it should be noted in your limitations.

The_Scary_Man wrote: »

Would it be acceptable from an ethical perspective to share the anonymised raw data?

It would depend upon the terms and conditions of your informed consent obtained from your subjects before data collection. Although data in aggregate may not generally reveal individual subject identities, one or more outlier subjects (if they occur) in the distribution may be unintentionally disclosed.