social twitter box blue 32social facebook box blue 32

 

Announcing: Project RED III Results are in!

The Project RED III results are in.  Over the next several months we will be releasing more in-depth insights into the findings, beginning with a planned event at ISTE.  Provided here is an initial summary of the findings, which provide the foundation for our further work in synthesizing and understanding the deeper meaning behind the results.

In 2010 our initial Project RED study demonstrated the non-negotiable imperative that proper implementation of education technologies was essential in raising student achievement and maximizing the return on investment. The results provided 9 Key Implementation Factors (KIFs) that most strongly influenced the results.  We further uncovered that through a full digital conversion, a large number of typical school district expenditures can be redeployed to offset the cost of implementing ubiquitous technology. Chapter 9 of the original research goes deeply into those discoveries.  Also significant was the finding that the lower the student to computer ratio, the better the return on investment.

In Project RED II we realized the need to examining the KIFs in action. Through a nationally competitive application process, we selected 20 Signature Districts based on their readiness, and their commitment to implementing the KIFs. From that group 17 districts remained committed to the 3 year program.

The goal of Project RED III was to independently analyze relationships between the KIFs, student achievement, and the overall return on investment. Project RED enlisted the support of the University of Memphis, Center for Research in Education Policy to analyze the year-over-year data provided by the Signature Districts, as well as publicly verifiable data such as state test score data, dropout rates, etc. Following is a 30,000 foot view of what they found, noting the disclaimer that, as is true with most research, some data was incomplete or not reported, and that questions emerge that will require further study.

ROI Analysis-Academics

Generally, the Signature Districts demonstrated gains academically and financially.  Five showed both positive financial and academic results.  The other twelve showed either some financial or academic improvements. 

Fourteen of the participating districts reported verifiable academic district data for Year 1 and Year 3, allowing for comparisons to be made in five ROI categories.  Four of the five areas showed promising academic results between years 1 and 3.

  1. Disciplinary Actions
    Eleven of the 14 districts, or 79% saw a decrease in the number of disciplinary action.

  2. Drop-out Rates
    Nine of the 14 districts, or 64% of districts saw a decrease in dropout rates.

  3. AP Classes Participation
    Eight of the 14 districts, 57% of districts saw an increase in the number of students taking AP classes.

  4. Graduation Rates
    Eleven of the 14 districts, or 79% saw an increase in graduation rates.

Dual Enrollment is the one success measure that showed a negative trend, with more districts reporting a decrease rather than an increase in participation. Although there were some reporting issues with the data in this category, we question if an increase in enrollment in AP courses may be affecting the number of participants in Dual Enrollment Courses.

ROI Analysis-Financial

Nine annual expenditure categories were examined. A third of the Signature Districts saw their overall expenditures decline. Of the nine categories studied, four showed the most promising results regarding decrease in expenditures between years 1 and 3.

  1. Printing Supplies
    When transforming learning through a 1:1 implementation, and operations through a district digital conversion, the need for printing should be reduced. This should reduce the expenditures for printers, printing supplies, and copy machines, as well as copy machine and printer maintenance as noted below.

  2. Copy Machine and Printer Maintenance
    See the response to number 1.

  3. Tech Expenditures
    We anticipated that technology expenditures would increase in all of the Signature District as they expanded their 1:1 programs. Surprisingly, these expenditures decreased in more than half of the districts.

  4. Textbooks
    Sixteen districts reported verifiable data. While nine of the 16, or 56% of districts reported decreases in textbook expenditures, 44% actually indicated an increase. The result seems to indicate that some of the Signature Districts are still struggling to make the shift away from textbooks.

When looking at the nine categories individually, five showed at least as many increases as decreases in expenditures between years 1 and 3. For example, an equal number of districts reported increases and decreases in expenditures for printers, and seven districts reported increases in copy machine expenditures. More investigation will be necessary to determine the specific factors that led to these results, but it is clear there is still room for growth in cost avoidance and reallocation of assets in all of the Signature Districts.

Academic Achievement Analysis

Generalized Findings

While the impact of the Signature District program was almost universally positive in increasing proficiency levels in mathematics and science, the proficiency levels in high school reading did not improve, and in some cases decreased. Basic reading skills, such as decoding, can be effective developed through online skill development software, but increased use of such technology carries the risk of replacing the conceptual thinking work needed for deeper understanding of text. What type of applications are used, and how they are applied may affect high school reading proficiency outcomes. As University of Memphis quoted in their report, “Reading comprehension is much harder to learn when someone (or something) else is comprehending the material for you” (Diamond & Mandel, 1996).  Therefore, to increase reading proficiency levels through the use of technology in the upper grades, technology must be used to encourage students to interact more deeply with the texts presented to them.

The University of Memphis also found that “mandating” the use of technology, with regular checks for compliance was not effective. There is, however, a connection between technology and increased academic achievement when the teacher uses technology more organically, and in innovative ways.

The increased use of Communication Tools, such as email or other digital platforms that students use to talk with the teacher and work with other students, showed a positive relationship with proficiency levels in elementary school science, middle school science, and high school mathematics.

High School Academic Findings

  • The use of technology and the increased use of change management strategies by the principal in high school mathematics and science showed a positive relationship with proficiency levels.
  • A written plan that includes systematic collection and usage of data had a statistically significant and positive affect on reading and mathematics scores.

Middle School Academic Findings

  • The use of technology, together with the increased use of change management strategies by the principal, showed a positive relationship with proficiency levels in mathematics.
  • The increased use of online formative assessments showed a positive and educationally meaningful relationship with proficiency levels of students in reading.
  • One consistent statistically significant finding for reading, mathematics, and science on the survey was that involving teachers in the planning of 1:1 professional development opportunities is not leading to improved academic performance in comparison to the state.

It is important to note that all of the statistically significant findings showed an inverse relationship between improvements the central office identified in the 1:1 program, and the actual academic improvements the middle schools showed on their state standardized tests. This may be an indication of a delay between improvements made on the district level, and the impact they have on the students in the classroom. It could also be that there is a disconnect between the perceptions of central office administrators, and the actual proficiency students demonstrated on their state standardized test.  

Elementary Findings

  • The increased use of online formative assessments showed a positive and statistically significant relationship with proficiency levels of elementary students in reading, mathematics, and science relative to the state. 
  • Increased use of search engines had a positive relationship with increased proficiency levels in reading, mathematics, and science relative to the state for elementary students. 
  • For the elementary schools, a statistically significant connection was found between the use of technology as a component of the teacher evaluation system, and increased proficiency levels in reading, mathematics, and science in comparison to the state. 

The reported use of technology in elementary school intervention classes did not show any measurable impact on academic performance.  This is in contrast to the findings of our original research. The type of technology intervention used, and the frequency of use can play a major role in its effectiveness. Therefore, it will be necessary look at what was implemented and how it was used in order to better understand the results. 

By Leslie Wilson, CEO of One-to-One Institute; and Michael Gielniak, Ph. D., COO of One-to-One Institute