Community colleges play a vital role in American society, helping millions of adults to achieve their academic and personal goals and preparing workers for the modern economy. Because of their low cost and accessibility, community colleges are especially important institutions for low-income students, students of color, and first-generation college students. Unfortunately, far too many students end up dropping out of community college without earning a certificate or degree or transferring to another college or university. While poor academic preparation and other challenges faced by students, such as having to work full time or being a single parent, are part of the explanation, policymakers are increasingly holding community colleges accountable for student performance and are looking for ways to help them increase student success.
Achieving the Dream: Community Colleges Count is a bold, multiyear, national initiative designed to help more community college students succeed, either by earning a certificate or degree or by transferring to another institution. The initiative is particularly concerned about helping students who have traditionally faced the most barriers to success, including low-income students and students of color. Launched by Lumina Foundation for Education in 2003, the initiative grew to involve 58 institutions in nine states by 2006 (see Figure ES.1). (Twenty-four colleges were added in April 2007, bringing the total to 82 institutions in 15 states.)
The initiative also involves many national organizations or foundations that play key supportive roles. Together, these organizations are working to change the culture and practices inside community colleges, as well as external factors that shape institutional behavior, such as public policy, research, and public engagement.
MDRC and the Community College Research Center (CCRC) are evaluating the work that is taking place inside community colleges — specifically, inside the first 27 colleges to join the initiative from Florida, New Mexico, North Carolina, Texas, and Virginia (called “Round 1 colleges”). The initiative is attempting to focus community colleges on understanding and making better use of data to improve student outcomes — a process that is referred to as “building a culture of evidence.” Participating colleges collect and analyze longitudinal data on student achievement along with other information that will help them identify factors that facilitate or hinder students’ academic progress. From these efforts, colleges are expected to assess what is happening on their campuses in an open, straightforward, and rigorous way and to make lasting changes in their operations and culture.
This report captures the progress the Round 1 colleges have made after one year of planning and one year of implementation of Achieving the Dream. (Altogether, the initiative will extend at least five years at the colleges.) The report also presents data on average institutional performance on selected student outcomes prior to the start of the initiative. Later on in the evaluation, these data will be used as a baseline against which changes in student achievement will be measured. The initiative’s design and this report’s principal findings are summarized below.
The Initiative’s Design
Most community colleges across the country gather large amounts of data on students through enrollment forms, placement tests, and academic transcripts. However, because community college funding is largely based on enrollments, there is little incentive for colleges to examine whether the same students return each semester or are accomplishing their academic goals in a timely fashion. Colleges that participate in Achieving the Dream agree to look at their data more closely in order to learn whether their students are staying in school and meeting other critical benchmarks, such as completing developmental courses and advancing to college-level courses. They also agree to break down their data to determine whether all students are making progress at the same rate or whether there are “achievement gaps” among some segments of the population, such as students of color or low-income students. Figure ES.2 depicts the theory underlying this work and how it is expected to lead to better student outcomes. The process unfolds in three stages:
- Diagnosis and planning. Colleges collect and analyze data to understand how students are faring over time and to identify any gaps in achievement among particular subgroups. If the analysis reveals that some students are not making adequate progress, colleges will be motivated to rethink existing practices.
- Implementation and evaluation. Colleges implement strategies to improve student performance, such as adopting new developmental education curricula or strengthening academic advising. They also conduct rigorous assessments to determine whether these strategies improve student outcomes.
- Institutionalization. Colleges adopt and expand effective strategies. Program review, planning, and budgeting are driven by evidence of what works best for students.
Achieving the Dream provides both financial and technical support to help colleges undertake this process. The financial support includes planning grants of $50,000 and implementation grants of $400,000 over four years that colleges can use to support data collection and analysis as well as implementation of program strategies. The technical support includes two outside consultants — a coach (usually a former community college president) and a data facilitator — who advise the college on how to perform the data analysis, interpret and communicate the findings to faculty and staff, and use the information to make improvements in college programs and services. The coach and data facilitator each spend 12 days working with the colleges during the planning phase and the first year of implementation, and they gradually reduce their time in subsequent years. Finally, all the colleges attend annual meetings where initiative goals are reinforced, promising program strategies are shared, and team-building is fostered.
Each college participating in Achieving the Dream is expected to commit its own resources as well. The president is indispensable in articulating a vision for student success and equity for all students. Teams of administrators, faculty, and staff are expected to take responsibility for the data collection and analysis as well as program planning, implementation, and evaluation. Ultimately, all college personnel have a role to play in ensuring student success. If the theory underlying the initiative is correct, over time more students are expected to complete developmental and “gatekeeper” courses (for example, English or Math 101), earn credits for the courses they attempt, persist in school longer, and earn more certificates and degrees. Gaps in achievement levels among various racial, ethnic, or income groups are expected to narrow or be eliminated.
Findings on Institutional Performance at Baseline
All the colleges participating in Achieving the Dream submit student records to a centralized database that was created for the initiative. Colleges were asked to analyze student records going back to 2002, three years before they received their implementation grants. Going forward, these data will be used as a baseline to determine whether colleges succeed in improving student outcomes.
Findings on institutional performance indicators at baseline are summarized in Figure ES.3. The statistics show what happened over a three-year period to first-time certificate or degree-seeking students who enrolled in one of the Achieving the Dream colleges in fall 2002. In this analysis, the average performance of every college — from the very biggest to the smallest — is weighted equally.
- The institutional measures show that, on average, students at the Achieving the Dream colleges are struggling academically.
Only a fraction of students at the Achieving the Dream colleges are reaching critical academic benchmarks. For example, between 29 and 37 percent of students who attempted the highest level of developmental math, English, or reading actually completed that course within a three-year period. About one out of five students referred to a gatekeeper math course are successful within a three-year period; for gatekeeper English, about three out of ten students are successful. Seventy percent of students who enrolled in fall 2002 stayed on for a second semester, but, by the third year, the enrollment rate for this group fell to 23 percent. On average, only about 11 percent of students who started in 2002 earned a credential or diploma within three years. Because these figures represent averages across Achieving the Dream colleges, success rates for students at individual colleges may be higher or lower. (The statistics for many other community colleges across the nation are similar.)
- There are gaps in achievement by racial and ethnic subgroups, but not on all measures. Sometimes the gaps run contrary to what was expected.
Across the Achieving the Dream colleges, the rates of persistence among African-American and non-Hispanic white students are roughly the same, while Hispanic students show significantly higher persistence rates on some measures. Unfortunately, persistence does not always translate to degree completion: The average rate of completing an associate’s degree was significantly lower for African-American, Hispanic, and Native American students than for non-Hispanic white students. Again, these findings are based on averages across all institutions; at individual colleges, achievement rates of various racial and ethnic subgroups may differ.
- Students who received Pell Grants (a proxy for low-income status) generally had better outcomes than students who did not receive Pell Grants.
The federal Pell Grant program provides financial aid to low-income college students. On average, the rates of successful completion of developmental and gatekeeper courses at the Achieving the Dream colleges were significantly higher for students who received Pell Grants than for nonrecipients. The average rates for earning credentials or being enrolled in college in the third year were also significantly higher among Pell Grant recipients than nonrecipients.
Findings on Implementation of Achieving the Dream
The evaluation team visited the first 27 colleges to join the Achieving the Dream initiative in spring 2006. At each college, administrators, faculty, staff, and others working on Achieving the Dream were interviewed, along with a few faculty members who were not directly involved in the initiative. The interviews focused mainly on the colleges’ efforts at problem diagnosis, planning, and early implementation of strategies for increasing student success.
Building a Culture of Evidence
- The “culture of evidence” concept resonated strongly with the colleges.
Nearly everyone interviewed at the 27 colleges supported the basic tenet of Achieving the Dream: to use data to diagnose barriers to student achievement and guide institutional decision-making on how best to serve students. Some interviewees spoke of a change in attitudes such that people were asking, “What does the evidence show?” as opposed to relying on personal beliefs or anecdotes. When confronted with data on poor student performance, interviewees at many of the colleges indicated that faculty and staff felt motivated to address those problems.
- Most of the colleges attempted to follow the general outlines of the data analysis process advocated by Achieving the Dream. Faculty and staff involvement was usually limited to a few individuals, however.
Achieving the Dream colleges were asked to create “data teams” to orchestrate the collection and analysis of student records and other information, such as student surveys. Most of the colleges did so, and the teams were usually headed up by each college’s director of institutional research. Some colleges also involved a few faculty and staff on the data teams. There were relatively few examples where the majority of faculty within academic departments (such as English and math) had begun looking at data.
- About half the colleges used data to identify priority problems to address during the initiative.
Many colleges reported using longitudinal data on student outcomes as well as focus groups and student surveys to identify priority problems that they wanted to tackle. There was disagreement among interviewees about the usefulness of the data that colleges were required to share with the initiative, which some individuals said was burdensome or duplicative of what their school was already doing. However, even these critics agreed that longitudinal tracking of students was a powerful method for identifying gaps in student achievement.
- Fewer colleges used their data analysis to select strategies for the implementation phase.
Achieving the Dream colleges are expected to choose programmatic strategies based on their analysis of data on students. Of the 27 Round 1 colleges, 10 were able to make links between the analysis they performed and the strategies that they chose to improve student performance. The remainder tended to base their selection of strategies on literature reviews, suggestions they heard at Achieving the Dream conferences, or ideas they had before becoming involved in the initiative.
- One out of four colleges had plans for evaluating the effectiveness of the strategies that they had chosen to improve student performance.
A fundamental premise in building a “culture of evidence” is that colleges should assess their programs and services to learn which strategies are working. Effective strategies should be preserved and expanded; ineffective strategies should be discarded. Most colleges had not yet thought about how to evaluate their program strategies. Among the colleges that did have evaluation plans, some suffered from weak designs, usually because they had not identified a valid comparison group.
- Two common obstacles to building a culture of evidence were difficulty retrieving and analyzing data from information technology systems and limited institutional research capacity. In addition, administration, faculty, and staff at most colleges were unfamiliar with how to use data to improve student outcomes.
Some of the colleges had computerized student record systems that were poorly designed or were undergoing conversions, which made it hard for them to perform the data analysis required by the initiative in a timely way. Over one-third of the colleges reported having insufficient institutional research capacity to support broad-based use of data for decision-making. Moreover, few administrators, faculty, and staff at the colleges were accustomed to using data in ways that Achieving the Dream advocates. On most campuses, the lack of familiarity with using data to drive decision-making extends to the most basic level, such as what questions to ask.
- Some faculty and staff were concerned that data would be used against them or were skeptical that increased use of data and research will lead to improved student outcomes.
At many colleges, at least a few faculty and staff felt that they were being blamed (or could be blamed) for poor student performance or that the outcomes being analyzed were not appropriate for assessing things that really mattered, such as whether students were learning the course material. Administrators at most of the colleges have tried to assure faculty and staff that the data are being collected to better understand student performance and make improvements and not to be punitive.
Leadership and Engagement
- College presidents expressed strong support for Achieving the Dream.
Presidents cited several reasons for wanting their college to participate in Achieving the Dream, including wanting to help students succeed, promote data-driven decision-making, and bolster reaccreditation processes. Several presidents reported that involvement in such a high-profile, national initiative added crucial external support and credibility to their desire to build a culture of evidence. Some presidents led or sat on Achieving the Dream committees, and many spoke about the initiative at college convocations and meetings with their governing boards.
- Management of the initiative was usually delegated to senior college administrators below the level of president.
Almost all the college presidents delegated responsibility for managing the initiative to other personnel, often to a vice president or a dean, although there were a few instances when the role was handed off to someone without sufficient stature. In the few places where this occurred, the initiative suffered as a result.
- One-third of the Achieving the Dream colleges recently experienced or were in the midst of a change in leadership.
Given the president’s key role in articulating a vision and building commitment for Achieving the Dream, a change at the top could slow down the progress of the initiative. A few colleges appeared to have this problem. In at least one case, however, a newly installed president was credited with bringing ideas and enthusiasm that the former president lacked.
- Most colleges had a core team of administrators, faculty, and staff who collaborated on the initiative and were enthusiastic supporters. Beyond this core team, faculty and staff involvement was generally limited.
Achieving the Dream expected colleges to create a core team of individuals to guide and manage the initiative. Most of the colleges did so, and the commitment of these individuals was high. At some schools, Achieving the Dream was credited with increasing communication among divisions of the college that formerly had minimal dialogue about student outcomes, such as student services and academic departments. Beyond the members of the core teams, however, the evaluators found that relatively few faculty and staff on most campuses were knowledgeable about Achieving the Dream activities at their colleges, though some had at least a general notion that there was a push to improve institutional performance or student outcomes.
- Some faculty and staff held beliefs or attitudes that ran counter to Achieving the Dream principles.
In addition to being a “data-driven” initiative, Achieving the Dream is founded on the idea that all students can succeed. At many campuses, at least a few faculty and staff expressed the view that students themselves should be held accountable for their outcomes — and not the institutions. Others blamed colleagues for poor performance or worried that efforts to increase graduation rates would mean lowering academic standards. Some administrators and faculty seemed troubled by the initiative’s particular emphasis on improving the success of students of color and of low-income students, believing that this amounted to favoritism or unequal treatment. Achieving the Dream has tried to promote the view that equity in outcomes does not imply treating all students the same; some students may need more support than others in order to succeed.
- Some of the colleges worked to incorporate student and community perspectives into their Achieving the Dream activities.
At almost half the colleges, student views — obtained through focus groups or surveys — were an important consideration in identifying key problem areas during the planning year. Incorporating student views into the decision-making process was a new development for some these schools. In general, however, the colleges did not involve students in the process of designing initiative strategies. Similarly, at least 11 colleges were working with other educational or civic organizations in planning and implementing Achieving the Dream strategies, but other colleges had not developed partnerships beyond their campus boundaries.
Strategies for Enhancing Student Success
- The colleges implemented a wide variety of strategies to improve student outcomes.
Across the 27 colleges, five prominent strategies selected to increase student success were (1) strengthening academic advising services; (2) creating or revamping orientation and “college success” programs or courses for incoming students; (3) supplemental instruction and tutoring; (4) learning communities, in which small groups of students take two or more linked courses together; and (5) professional development, including training in cultural competence and racial dynamics for faculty and staff.
- By spring 2006, just one year into the Achieving the Dream initiative, the majority of colleges had begun piloting or implementing at least one of their strategies.
Most colleges tried to work on several fronts to improve student success — for example, strengthening advising services and developing learning communities. They often focused on one or two strategies initially and planned to try additional approaches in the coming academic year.
- Many of the colleges’ strategies had only been partially implemented and affected relatively few students.
Where the evaluation team was able to assess how many students were affected by the colleges’ strategies, the numbers generally were small. Only a minority of strategies targeted all students. Some colleges were sufficiently advanced in their efforts to be characterized as having reached full implementation, though in most cases these strategies were built on programs that predated Achieving the Dream or were one-time events, like diversity training for faculty and staff.
Early Progress Toward Institutionalizing the Principles and Practices of Achieving the Dream
Given that the 27 colleges were just nearing the end of their first year of implementation grants when field visits were conducted — and had three more years to go — the evaluation team did not expect to find that most colleges had fully institutionalized Achieving the Dream principles and practices. Nonetheless, the team looked for early signs and grouped the colleges according to their progress.
- Six colleges already showed clear signs of institutionalizing a culture of evidence. Five more had taken important preliminary steps in this direction.
Achieving the Dream differs from most grant-funded initiatives in its focus on institutional change. Although it is too soon to make definitive conclusions, six colleges showed strong indications of broadly engaging administrators, faculty, and staff in using data for program review, strategic planning, and budgeting. Another five colleges had adopted, or were in the process of adopting, evidence-based strategic planning procedures. It is important to note that nearly all these institutions reported that they were moving in this direction before they became involved with Achieving the Dream, although they also indicated that the initiative energized and helped focus their efforts.
- Ten colleges had in place some of the building blocks of a culture of evidence. Six colleges were struggling.
Ten of the remaining 16 colleges had gathered and analyzed data to identify gaps in student achievement, though the connection between their analysis and the strategies being implemented was not always clear. Some of them were hampered by difficulties retrieving and analyzing data — usually because of weak or cumbersome information technology systems or overburdened information technology or institutional research departments — or by turnover of key leadership. Six colleges had limited data collection and analysis capabilities and had not figured out how to begin using data on student outcomes to evaluate and improve programs and services.
- Accreditation processes and state policy initiatives helped to reinforce the goals of Achieving the Dream at a majority of the colleges.
Interviewees at 23 of the colleges mentioned the connection between Achieving the Dream and the increased emphasis by accreditation agencies on using data about student outcomes to guide decision-making. In Texas, interviewees at two colleges mentioned the synergy between the goals of Achieving the Dream and those of the state’s “Closing the Gaps” initiative, which seeks to address growing inequities in college access and attainment among the state’s growing Latino population. Virginia’s community college system developed a strategic plan that explicitly referenced Achieving the Dream and the use of data to revise policies.
Conclusion
Achieving the Dream is attempting to change the practices and cultures of community colleges by focusing them on using data to diagnose problems and make lasting improvements in institutional practices and cultures. This report finds substantial evidence that most of the first 27 colleges to join the initiative have fully embraced the goals of the initiative. A subset of six colleges already appears to have begun to institutionalize the principles and practices of Achieving the Dream, in that they have involved a broad segment of the campus community in analyzing data on student outcomes and were using data for strategic planning, budgeting, and review. Another five colleges were taking preliminary steps in this direction. The other 16 colleges — while making some progress — often had difficulty meeting the initiative’s research demands. Many of them faced problems retrieving data from student information systems or had limited institutional research capacity. Most lacked experience or know-how in using data to inform improvements in programs and services. Most had yet to involve faculty and staff outside the core team. A few colleges lacked strong attention from their leadership or experienced turnover in key administrative positions.
Nearly all the colleges had begun to implement program strategies to improve student success. In 10 colleges, the strategies were clearly linked to the data analysis that they had performed; in the remainder, program strategies were selected based on a review of the literature, presentations at Achieving the Dream conferences, or ideas that college personnel had been contemplating before becoming involved with the initiative. As would be expected at this early stage of the initiative, most of the program strategies were just getting off the ground and affected relatively small numbers of students. A majority of colleges had not yet developed plans to evaluate the effectiveness of their chosen program strategies.
MDRC and CCRC will return to the colleges in 2009 to conduct a second round of interviews with college personnel and will administer two rounds of surveys to college administrators and faculty in 2007 and 2009. The purposes of the interviews and surveys will be to determine how the initiative has evolved and to capture changes in institutional practices and attitudes. At a subset of colleges, MDRC will also assist in conducting a rigorous assessment of programs or strategies that have been implemented to improve student outcomes. At other selected institutions, there will be an in-depth study on the cost of Achieving the Dream and an ethnographic study of students. A final report is planned for 2010.
|
