Education Research

Below are summaries of studies on VIP, which may be of use to faculty writing proposals.


Student Leadership Development over Timechart: student technical leadership by academic rank and number of semesters on the team, as rated by peers..

Takeaway: Leadership activity as rated by peers increases with number of semesters in VIP through the third semester. Leadership within VIP teams is not correlated with academic rank, which means a typical sophomore and typical senior will provide comparable levels of leadership (coordination and technical leadership) in their first semester of VIP. Findings support observations that returning students help lead their VIP teams.

Scope: Mean ratings on two peer evaluation questions (N = 1,073 and N = 1,047) in Spring 2021 by student academic rank and number of semesters of participation in VIP.

This study explored student leadership development over multiple semesters. The study examined peer evaluation ratings in general leadership (coordination of teams’ work) and technical leadership (serving as a technical/content area leader) in a single semester VIP. Analysis examined means and distributions on two peer evaluation questions (N = 1,073 and N = 1,047) by student academic rank and number of semesters of participation in the program. Within their teams, students’ leadership increased through the third semester, with students making their greatest leadership contributions in the third semester and beyond. Students of lower academic rank provided as much leadership (including technical leadership) as older students who had comparable experience on the team. The study was novel, because the measurement of student leadership was based on peer-evaluations instead of self-assessments.

Citation (IEEE): J. Sonnenberg-Klein and E. J. Coyle, “Experience vs. Academic Rank: Leadership Growth over Multiple Semesters in Vertically Integrated Project (VIP) Courses,” in 2022 IEEE Frontiers in Education Conference (FIE), Oct. 2022, pp. 1–5. doi: 10.1109/FIE56618.2022.9962651.


Unpublished Report: Student Gains in Job Placement, Interest in Gradaute School

The Office of Academic Effectiveness found gains in job placement and graduate school interest for VIP participants. The study requires further research on ways in which different subgroups benefitted, but the findings are available to VIP Instructors.  To access them, please login (instructions below).

  1. Click here to login, or click the person button in the far upper-right corner.
  2. Once on the login page, scroll to the bottom of the page and click the "Log In" button, as shwon in the screenshot to the right.  This will take you to a GT login page.
  3. Once you are logged in, a new menu item will appear in the "Current Instructors" menu named "For Studies on VIP Part 2 (Now Viewable)."

 


Student Gains: Ability to work in multidiscplinary teams; Ability to work with individuals from diverse backgrounds; Understanding of technology applications relevant to field of study

Takeaway: In an analyis of GT exist surveys, VIP students agreed more strongly than non-VIP participants that their GT educations contributed to their ability to work in multidisciplinary teams; ability to work with individuals from diverse backgrounds; and understanding of technology applications relevant to their field of study.

Scope: Compared GT exit surveys for VIP participants (N=233) and non-participants (N=1,781)

Georgia Tech administers exit surveys to undergraduates prior to graduation.  The survey includes questions on how the students’ experiences at Georgia Tech contributed to specific skills and abilities.  VIP staff identified seventeen questions related to typical VIP program activities.  The Georgia Tech VIP program was established in Spring 2009.  At the time of the analysis (autumn of 2015), 384 VIP participants had graduated, and 233 had completed the exit survey.  The GT Office of Assessment created a comparison group of non-VIP participants using stratified random sampling, and performed a t-test to compare survey results for the two groups.  For the seventeen questions, VIP participants had generally higher ratings than non-VIP participants.  The difference was statistically significant on six questions:

  • Understanding of technology applications relevant to your field of study
  • Ability to work with individuals from diverse backgrounds
  • Ability to work in a multidisciplinary team
  • Ability to carry out projects independently
  • Development of leadership skills
  • Overall career training
     

Of the six questions, there was a meaningful effect size (Cohen’s d of .2 or greater) on three questions: To what degree did your Georgia Tech education contribute to your:

  • Ability to work in a multidisciplinary team (t(1982) = 4.437, p < 0.001, d = 0.313);
  • Ability to work with individuals from diverse backgrounds (t(1987) = 3.271, p = 0.001, d = 0.231);
  • Understanding of technology applications relevant to your field of study (t(2002) = 3.19, p = 0.001, d = 0.224).
     

These results motivated the more recent analysis of peer evaluations, to look at student interaction by discipline and background (race/ethnicity).

Citation (IEEE): J. Ludlum, “Georgia Tech Exit Survey Research Brief: VIP Program Comparisons,” Georgia Institute of Technology, 2015.


Within VIP, students interact more often with students of other races/ethnicities and more often with students from other majors

Takeaway: Within VIP, students interact more often with students from other races/ethnicities than their own and more often with students from majors other than their own.

Scope: GA Tech peer evaluations, Spring 2016 (N=355)

Summary: Seeing value in Melkers’ social network analysis of VIP teams, the GT VIP program included social network questions in peer evaluations.  We conducted an analysis of Spring 2016 peer evaluations.  External-Internal (EI) indexes indicated that on average, students interacted more often with students of other races/ethnicities than their own and more often with students from other majors.  Quadratic Assignment Procedure (QAP) analysis (which accounts for team composition) showed that of the twenty-four teams: one had significant correlation between race and student interaction, and clumping was was clear on visual inspection; six had significant correlation between student major and interaction (clumping by major was not unexpected).  While the EI index showed clumping by gender and domestic/interantional status, the QAP analysis accounts for team composition and did not show clumping on these factors.  The clumping on one team by race/ethnicity led to the development of additional professional development for instructors.

Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, E. J. Coyle, and H. H. Ai, “Multidisciplinary Vertically Integrated Teams: Social Network Analysis of Peer Evaluations for Vertically Integrated Projects (VIP) Program Teams,” presented at the 2017 ASEE Annual Conference & Exposition, 2017.

Full Paper


Student Leadership Development: Returning students provide more peer-help and peer-management, regardless of academic rank

Takeaway: Within VIP, returning students provide more peer-help and peer-management regardless of their academic rank.  This confirms that returning students take on leadership roles within their teams, both in terms of providing help and team management.

Scope: GA Tech peer evaluations, Spring 2017 (N=484)

Summary: Analysis of Spring 2017 peer evaluations looked at peer support (giving help) and peer management by major, academic rank and status as new or returning VIP students.  External-Internal (EI) indexes showed more peer support between students of the same major, but when team composition was accounted for in Quadratic Assignment Procedure (QAP) analysis, much of this correlation fell away.  There was significant correlation between major and giving/receiving help (clumping by major) on 5 (15%) of the 34 teams studied.  For 29 (85%) of the 34 teams, major was not correlated with giving/receiving help, representing a balance of peer-helping across disciplinary lines.

A UCINET version of analysis of variance was used to examine student rank with respect to giving help and management ratings, as well as student status as new or returning students.  Student rank did not show statistical significance in who students went to for help or how students rated teammates on management ability.  In contrast, student status as new or returning VIP students was statistically significant on both measures, with returning students rated higher.  This shows that the social capital within teams is tied to students’ prior experience on the teams, and not necessarily their academic rank.  The findings also confirm that returning students take on leadership roles within their teams, both in terms of providing help and in team management.

Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, and E. J. Coyle, “Social Network Analysis:  Peer Support and Peer Management in Multidisciplinary, Vertically Integrated Teams,” presented at the Manuscript submitted for publication, 2018.

Full Paper


Equity

Takeaway: Nationally, Black/African American and Hispanic/Latinx students (historically underserved, HU) participate in undergraduate research at lower rates than their peers.  The GT VIP Program enrolls representative proportions of HU students with respect to the campus population.  After their first semester of VIP, students enroll for second (and more) semesters of VIP at the same rate, regardless of race/ethnicity, representing equity in student persistence in the program.

Scope: Analysis of student enrollment and number of semesters in VIP with respect to race/ethnicity (N = 1,251)

While historically underserved (Black/African American and Hispanic/Latinx) students derive greater benefits from undergraduate research experiences with with faculty, across the nation, historically underserved students participate at lower rates than their peers. In contrast to the national trend, Georgia Tech’s VIP Program enrolls representative proportions of Black/African American students and Hispanic/Latino students with respect to the campus population. An analysis of enrollments over time looked at the number of semesters students enrolled in VIP with respect to race/ethnicity.  Findings indicate that students of different races and ethnicities persist in the VIP course sequence at equal rates.

Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, and E. J. Coyle, “Diversity and Student Persistence in the Vertically Integrated Project (VIP) Course Sequence,” presented at the 2018 Collaborative Network for Engineering and Computing Diversity (CoNECD), 2018.

Full Paper


Policies on how VIP Credits Count toward Degree Requriements Affect Student Return-Rates

Takeaway: There is a correlation between department policies on how VIP credits count toward degree requirements and how many semesters students participate in VIP.  Two policy elements correlate with higher persistence: 1) allowing all VIP credits to count as in-major electives after a minimum number are earned; and 2) allowing students to fulfill culminating design requirements through VIP, with no additional planning/requirements beyond the normal design sequence.  Matches and mis-matches between student major and instructor home department are not correlated with student persistence in the program.

Scope: Analysis of Persistence in the VIP Course Sequence with respect to Major (two analyses, N = 869, N = 45)

A key to launching and maintaining productive VIP teams is having students participate for multiple semesters. This enables deeper learning experiences and provides the time for students to develop deeper expertise and take on increasing levels of responsibility. Academic departments have established a range of credit-use policies to incentivize their students’ multi-semester participation, with different incentives and varying thresholds for incentives.  Other factors that may affect student persistence are matches/mismatches between students and their instructors’ departments, and student academic rank in their first semester of VIP.  This study profiles policies for the four largest academic units participating at Georgia Tech and examines student persistence (N = 869) by department, by academic rank, and by matches-mismatches between student and instructor departments.  In a secondary analysis, persistence rates are compared for a degree program before and after an incentivizing credit-use policy was established (N = 45). Results indicate two elements correlate with higher persistence: 1) allowing all VIP credits to count as in-major electives after a minimum number are earned; and 2) allowing students to fulfill culminating design requirements through VIP, with no additional planning/requirements beyond the normal design sequence.  Matches and mis-matches between student major and instructor home department are not correlated with student persistence in the program.  This indicates that instructors leading these multidisciplinary teams are equally enageing students from outside their own departments.

Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, and E. J. Coyle, “Correlation between Academic Credit-use Policies and Student Persistence in Vertically Integrated Project (VIP) Courses,” presented at the 2018 ASEE Annual Conference & Exposition, 2018.

Full Paper


Learning gains for new and returning VIP students

Takeaway: Students who sought help from other students reported higher learning gains.  Returning students reported greater gains in technical skills, team skills and some communication skills.  Higher enthusiasm was correlated with greater gains.

Scope: GA Tech Purdue VIP Programs; N=98 first semester; N=180 second semester.  Another campus was initially included, but their numbers were too small to include in published results.

Summary: In Fall 2010 and Spring 2011, an evaluation of VIP was conducted by Julia Melkers (Melkers et al., 2012) on social networks within teams and between teams, and on students' self-reported learning.  The study looked at social networks and examined self-reported learning outcomes for students who provided help to teammates; students who sought help from teammates; new students and returning students; and student levels of enthusiasm. The analysis included social network analysis and regression analysis.  The social network analysis showed students were interacting within teams, and seeking/receiving advice in different areas (technical, managerial, other).  In self-reported learning gains, outcomes for students who provided help to others did not differ, but learning outcomes for students who sought help from others was statistically higher, both in technical skills and managerial skills.  In comparing learning gains for new students and returning students, results indicated greater gains for returning students in technical skills, in team skills and in some communication skills.  For students with higher levels of enthusiasm, results indicated greater gains across various skills, as would be expected.

Citation (IEEE): J. Melkers et al., “The Social Web of Engineering Education: Knowledge Exchange in Integrated Project Teams,” presented at the ASEE Annual Conference and Exposition, 2012.

Full Paper