Brief Description

Students use one-way between-subjects analysis of variance to answer a research question using a personalized data set. The students submit a formal APA style results section with appropriate tables, figures, and statistical tests to help support their findings. The assignment is designed to assess specialized knowledge, quantitative fluency, communicative fluency, and applied and collaborative learning.

Background and Context

The assignment is designed for students registered in an inferential statistics course that covers Analysis of Variance. This assignment may be presented in undergraduate level or graduate level courses. It may be part of a sequence of assignments that use the same dataset or a standalone assignment. It may be extended to be a project or used as a weekly assignment.

This assignment is a unique learning opportunity for students. Each student is provided the same introductory prompt and tasks to complete. All of their results, however, are distinct. Students are each assigned a simulated dataset. The datasets contain the same number of individual observations and variables. They do not, however, contain the same values for each observation. Each dataset is simulated by the instructor. This allows the instructor to create data that elicits specific results (i.e. model fit/lack of model fit, validation/invalidation of assumptions, evidence of a significant/lack of evidence of significant effects, etc.). Simulating data provides a sense of control to the instructor unlike trying to find real data to help facilitate learning of statistical topics.  As many statistics instructors experience, it can be difficult to find a real dataset exactly as desired to teach specific topics. That problem is overcome when data is simulated.

On the first day of the course each student is presented with their assigned dataset. For example, John Doe is assigned DataSet23. The students will use this dataset for six subsequent assignments. The students know this data has been simulated. Be that as it may, they become immersed in the material. For example, one student inquired about publication of their results before being kindly reminded that the results were simulated. Students feel ownership of their data. Consequently, they speak passionately about their work.

For this assignment, students are required to submit a formal written report in APA (American Psychological Association; 6^th Edition) style (subsequently referred to as a Write-Up). They are encouraged to work with their classmates when importing the data, running the analysis, and interpreting the results. Students may obtain differing conclusions due to randomization during simulation. Please note that randomization can be controlled by the instructor when simulating data. Datasets that result in the same conclusions may also be simulated. The randomization resulting in disparate conclusions is designed to enhance learning. For example, students working in pairs will not confirm their answers with classmates. They must engage in discussion about the results. They are not able to acquiesce with their classmates’ work. Examination of and discussion about each’s work must take place.

The assignment was piloted with first year Master’s degree students and first year doctoral students. Master’s students were in the school of psychology, psychological sciences, clinical and school psychology, counseling and supervision, communication science and disorder. Doctoral students were majoring in strategic leadership and assessment and measurement. Students were mixed among two parallel sections of an Intermediate Inferential Statistics course. Topics covered in the course were sampling distributions, t tests, correlation, regression, power, One-Way and Factorial Between – Subjects ANOVA, One-Way and Factorial Within-Subjects ANOVA, mixed ANOVA, and an introduction to ANCOVA.

The assignment is mapped to the following student learning outcomes (SLOs):

After completion of the inferential statistics course, students will be able to

• Make inferences using analysis of variance.
• Test the statistical assumptions underlying Analysis of Variance.
• Use SPSS to analyze data from one-factor ANOVA designs containing between-subjects factors.
• Write-up results to analysis performed using ANOVA in APA style.
• Test a set of post hoc comparisons for significance.

Please note that this particular assignment is not the only learning opportunity mapped to these specific SLOs.

Alignment and Scaffolding

At the time of assignment, students should have several foundational skills:

• Students must understand SPSS basics. This includes importing data, using the Point-And-Click System, generating descriptive statistics, developing elementary graphics (i.e. histogram, box plots), and performing introductory inferential tests (i.e. independent t test).
• Students have prior experience with one-way between-subjects ANOVA content in-class and outside-class. In the pilot study, students had two weeks of engagement with one-way between-subjects ANOVA content prior to the assignment. Students were introduced to the concept of ANOVA during the first week. They were assigned practice that covered the breadth of the topic. This ranged from mathematical problems by hand to conceptual meta-cognitive checks for reinforcement. During the second week, students were provided more details of ANOVA including an hour and fifteen minute lab session. The lab session was split between instructor led demonstration and Think, Pair, Share (Lyman, 1981) activities. After the in-class lab exercises, students were permitted one week to complete the assignment.
• Students should have experience writing APA style prose. They are not expected to have mastered this skill at the date of assignment. APA style, however, should not be a brand new concept as initial familiarization would require significant time commitment.
  • To learn more about APA style please refer to:

American Psychological Association. (2010). Publication manual of the

American Psychological Association. Washington, DC: American Psychological Association.

Nicol, A. A. M., & Pexman, P. M. (2010a). Presenting Your Findings: A

Practical Guide for Creating Tables.  Washington, DC: American Psychological Association..

Nicol, A. A. M., & Pexman, P. M. (2010b). Displaying Your Findings: A

Practical Guide for Creating Figures, Posters, and Presentations. Washington, DC: American Psychological Association.

The assignment presented here is the fourth in a sequence of six total assignments. The initial assignment presents an opportunity for students to become familiar with SPSS and their assigned dataset. They are presented with several questions that require the labeling of variables and values, variable manipulation and creation, the production of descriptive statistics, and the generation of graphics. Additionally, they are required to produce a short APA style paragraph describing the sample’s major demographics. This participant characteristics paragraph will be used for subsequent assignments.  The second assignment requires students to draft their first formal results section answering two research questions using t tests. The third write-up requires students to answer two research questions related to correlation and simple linear regression. This assignment is the fourth assigned and is the first regarding analysis of variance. The fifth assignment requires students to produce a write-up for an example of factorial between-subjects ANOVA. The sixth and final write-up during the one semester course requires students to perform a one-way within-subjects ANOVA.

The datasets are simulated to contain information that could be used to answer research questions regarding two way factorial within-subjects ANOVA, ANCOVA, and the general linear model. By the end of the semester, students will have significant practice running analyses in SPSS as well as translating and interpreting the results into an APA-style results section. These practical skills will then be used when students read research journals and/or perform their own research.

Reflections

The assignment has been piloted with 33 intermediate inferential statistics students at the Master’s and doctoral level. The course is titled intermediate; however, students may have taken their introductory statistics course 5 – 10 years prior at their undergraduate institution. As a result, the first two weeks of the course are taught as an expedited introductory course. For additional review, it is recommended that students participate in the Khan Academy Statistics and Probability section.  In this section, students may watch instructional videos and complete metacognition quizzes (https://www.khanacademy.org/).  Achievement, as measured by a percentage, was as expected. Grading was generous as the assignment was a brand-new type. For example, there was some degree of academic freedom allowed; not all write-up assignments were required to read the same. Students were only penalized if they did not include required content: an introductory paragraph, descriptive statistics, assumption checking, results of the omnibus test, post hoc tests, and a conclusion. Students without significant omnibus tests were penalized for including post hoc tests; students with significant omnibus tests were penalized for not including post hoc tests.

Feedback from all students was not elicited by the instructor. Informal anecdotal e-mail and in-person verbal communications were received from a few students. On the first day of the course, when the simulated datasets were explained, student rumblings of “Wow, that’s cool”, “That’s different”, and “Seems interesting” were overheard by the instructor.

After this particular write-up was assigned, students regularly commented on the enjoyment and uniqueness of the assignment. They specified elation using their “own” datasets. The word “own” was commonly used by students. It gave an appearance of a sense of ownership. A sense of responsibility and partnership with their data was cultivated over the course of the semester as more assignments were completed.

During office hours, two separate students conveyed their experience with the assignment in detail. Those discussion are summarized below. Terms in quotations were used by the students. The first, a first year Master’s student spoke to the unique challenges of an assignment of this type. Disregarding term projects, no other class had multiple assignments with “personalized” data sets. This was also their first experience in being posed a research question one would encounter in an “actual” study. They expressed the challenge of not knowing expectations when answering “a real research question using real data without step by step instructions”. When pressed about their confusion about expectations, they admitted the assignment was a “nerve wracking experience working with data no one else had”. Specifically, they noted that not being able to compare their work with that of fellow classmates was stressful because they didn’t know if they had the “right answer”.

The second student, a first year doctoral student, expressed enthusiastic gratefulness towards the “fun” experience. Contrary to the first student, this student was happy that their results were different from those of their best friend in the class. They specifically noted that it wasn’t just “different numbers, but, different conclusions” as well. He noted that they engaged in discussion about whether or not they had done something incorrect─specifically, asking why their results differed. He, somewhat perturbed, felt he did the assignment twice─once with his data and once with his classmate’s.

The conversations with both students were insightful. The second student’s comments particularly struck me. A desire I had, when designing the assignment, was to elicit students’ independent work followed by a group discussion. Originally, I debated about facilitating group discussion in class about the assignment the day it was due. Unfortunately I was unable to have in-class discussions due to time constraints. It is clear, through discussion with the second student, that this was achieved (at least for some).

In office hours, students, in general, were asking the correct questions. Students sought clarification on how to interpret what the results were indicating, what they were not indicating, etc. I do have some hesitation of proceeding with this exact iteration of the assignment. For example, a handful of students expressed frustration with APA style. This was an unintended consequence of the assignment. Developing comfort using APA style is important, but perfection is not an essential student learning outcome.

Student feedback about the collection of assignments were elicited through end-of-term student evaluations.  At the end of the pilot semester, questions regarding these write-up assignment types were posited to students.  On the scale, 1 – Very Well, 2 – Well, 3 – Neutral, 4 – Poorly, 5 – Very Poorly, students averaged 1.25 (standard deviation = .5) when asked how well the assignments reflected the current content and emphasis of the course.  When asked how much the formal APA style write-up assignments contributed to their understanding of the course material, students responded, on average, 1.44 (standard deviation = .79) on the scale 1 -A great deal, 2 – Much,  3 – Somewhat, 4 –  Little, 5 – Not at all.  In addition to Likert type responses, anonymous student testimonials included:

“The assignments were very helpful, particularly the write-up assignments.  I feel like they were a perfect representation of what I was hoping to get out of this course – practical statistics skills I use to analyze and present my own data.  Examples of APA style was extremely helpful.”

“The assignments were super helpful for giving me the opportunity to apply and internalize the skills necessary for this class and applying statistics to research and scholarship.”

“The assignments were super helpful.  At times the assignments were challenging enough to really facilitate learning.  I believe they were really helpful.”

“Write-ups were good, and kind of fun to apply what we learned in class to real problems.”

Future iterations of this assignment will include a detailed rubric and grading criteria guideline. For the pilot study, the assignment was graded holistically. Students were penalized one point for each missing required element or each incorrect element. Students were penalized one point for incorrect formatting and one point for poor grammar. Grades ranged from 88% to 100%, with one student failing to submit the assignment receiving a 0%. The course is designed with assignments to be learning tools. Students who demonstrate effort to successfully complete the assignment typically receive grades between 85% and 100%.

This write-up is part of a series of six assignments spread out over the course of the semester. Students have repeatedly thanked me for creating a series of connected assignments. The idea, in the future, would be to build these assignments into a semester-long project culminating in a final report. Students have shown significant appreciation for having their “own” simulated datasets.

For Assignment Library users, simulated datasets are available upon request. I know, however, that class sizes vary. If more datasets are needed, please do not hesitate to ask. One advantage of this assignment is that it may be adapted to any report style (APA, MLA, etc) and any Statistical Software Package (SPSS, SAS, R, MINITAB). It may be used as a standalone assignment or part of a series of connected assignments. It may painlessly be extended to research questions regarding One-Way/Two-Way Between/Within-Subjects ANOVA, General Linear Models, etc.

Rubric(s)

1. Sample Shared with Students
2. Examples of Completed Student Work