Lead Evaluator: Joan Karp
Associates: Samara Hoyer-Winfield, Elizabeth Osche
Primary partners: University of Massachusetts, Boston (UMB), Northeastern University (NU), Boston Public Schools (BPS)
Funder: National Science Foundation—Math Science Partnership (MSP)
Abstract: The Boston Science Partnership aims to strengthen BPS middle and high school science education, primarily by raising teacher quality, in order to increase student achievement, the number of students taking higher-level science courses, and students entering science, technology, engineering, and math (STEM) higher education programs. Its goals also include improving university-level teaching, training STEM faculty to be knowledgeable partners in science education reform efforts, and creating institutional changes at the universities that will ensure a continuation of support for and involvement with K-12 science education. The project takes a comprehensive approach in order to tackle major institutional barriers to achieving its goals, as well as to test out specific strategies.
The strategies that BSP utilizes to meet project goals are:
- Contextualized Content Courses (CCC)—courses taught by teams of STEM faculty and BPS teacher leaders that provide challenging content courses for science teachers in the context of the Boston Public School’s curriculum and district-approved pedagogical practices
- Collaborative Coaching and Learning in Science (CCLS)—BPS professional learning communities that provide high-quality science professional development within the context of each school
- Vertical Planning (VP)—PD sessions that bring together STEM faculty with K-12 teachers to build a greater understanding and better alignment within the entire science curriculum
- Support for students, teachers, and higher education faculty—this includes an extensive AP support program of activities, STEM seminars on teaching and learning at the university level, focus on issues of tenure and promotion, and inclusion of STEM faculty at community colleges in STEM pipeline efforts
- Partnership between BSP, UMB, and NEU
PERG has taken a 2-pronged approach to the evaluation questions for BSP:
- Project Goals: Has BSP achieved its primary project goals? How? What has been the impact of each of the primary strategies on achieving these goals?
- Project Strategies: What is the design, implementation, quality, contribution of each strategy? What are the benefits, issues, lessons learned, sustainability factors, and impacts?
Lead Evaluators: Judah Leblang and Joan Karp
Associates: Susan Cohen, Elsa Bailey
Funders: National Science Foundation (NSF) and the National Aeronautics and Space Administration (NASA)
Abstract: Cosmic Questions is an interactive 5000 square-foot exhibition, developed by staff at the Harvard-Smithsonian Center for Astrophysics (CfA), to promote reflection about and interest in “big questions” about the universe and humanity’s place in the cosmos, along with providing the most up-to-date information about the universe. Through their experiences in the exhibit gallery and at a series of related activities—a a short play, a staff demonstration, and a planetarium show—visitors are encouraged to construct meaning and to find relevance in relation to their own world views and life experiences.
Cosmic Questions was framed around three “big questions:”
- “What is the universe like?”
- “Was there a beginning to time?”
- “Where do we fit in?”
In this context, the exhibition had four major goals for its viewers, as described in project literature:
- Learn about key astronomical and scientific concepts, including:
- The composition of the universe and its vast scales of space and time
- The physical and analytical tools of the astronomer; learning from light
- The interplay of models, evidence and explanation in forming our understanding of the universe
- Increase their understanding of the nature of scientific inquiry by engaging in activities that explore “how we know” about the universe.
- Encounter various human perspectives (historical, personal, cultural, artistic, etc.) on age-old cosmic questions.
- Reflect upon their own ideas about the universe and the meaning and relevancy of the ongoing human search for answers to cosmic questions.
Evaluation Activities and methods:
PERG’s evaluation consisted of 3 phases. In 2000, evaluators conducted a series of front-end interviews at Boston’s Museum of Science (MOS), to determine visitors’ interest in and reaction to questions such as: “Are you interested in new discoveries about the universe?” and “Do you have a mental picture of the universe?”
Based on our findings, the exhibit developers created a scaled-down prototype exhibit at the Museum of Science in spring 2001. PERG evaluators collected extensive data at the MOS, and produced a formative report, which proved useful in the creation of the full exhibition.
The summative evaluation of Cosmic Questions was conducted between from September 2002 to June 2003. The evaluators collected data at 2 sites: Boston’s Museum of Science and the Midland Center for the Arts in Midland, Michigan.
Evaluation methods included:
- Exit interviews with museum visitors
- Follow up phone interviews about 1 week after the museum visit
- Survey of visitors who attended the related play “Boy Meets Girl”
- Survey of visitors who attended the related planetarium show
- Interactive observations, in which evaluators accompanied visitors on their trip through the exhibition
Key Findings: Visitors to Cosmic Questions generally found the exhibit highly engaging. They reported learning new information about the universe, and reflected on the ‘big questions’ embedded in Cosmic Questions. Almost two-thirds of our respondents said they had questions stimulated by the exhibit, and more than 60% said Cosmic Questions helped them think about their place in the universe.
Report: [Attach executive summary]
Lead Evaluator(s): Carol Baldassari and Sabra Lee
Associates: Judah Leblang, Samara Hoyer-Winfield, Elizabeth Osche
Partners: Boston University (BU); the Education Development Center, Inc. (EDC), a non-profit research and development organization; 5 Massachusetts school districts located in the Greater Boston area (Arlington, Chelsea, Lawrence, Waltham, and Watertown); and, as supporting partners, the University of Massachusetts Lowell (UML) and Worcester Polytechnic Institute (WPI) and the Program Evaluation and Research Group (PERG).
Funder: National Science Foundation – Math Science Partnership
Years: 2003- 2009
Abstract: Focus on Mathematics is a 5-year, grades 5–16 Targeted Math-Science Partnership program that offers a multi-faceted professional development program for mathematics teachers in grades 5-12.
Five (5) high schools, 16 middle schools, and 11 elementary schools from the 5 districts are involved in the FoM program. Over 200 mathematics teachers at the middle and high school levels (grades 6–12), as well as some districts’ grades 4 and 5 teachers, participate in FoM professional development opportunities. Together, the middle and high schools in these 5 districts serve almost 15,000 students.
FoM is based on a specific vision/hypothesis that informs the design of their professional development program for teachers, which is that deep immersion in mathematics leads to the development of teachers’ knowledge of mathematics for teaching. The professional development program, called the Professional Academy, was designed to provide a comprehensive set of ‘immersive’ learning opportunities where participants engage in mathematics at a deep conceptual level. Middle and high school teachers from the 5 district partners work alongside mathematicians and mathematics educators in the various professional development activities. The Academy includes a Master of Mathematics for Teaching (MMT) degree and the Professional Development Portfolio—week long summer institutes, school-based teacher study groups led by mathematicians, on-line courses, after-school seminars, and colloquia.
PERG’s evaluation combines qualitative and quantitative data collection methods to assess—
- The quality of the FoM professional development programs for teachers
- The ways in which teachers increased their knowledge of mathematics for teaching, changed their instructional practice, and provided leadership in mathematics;
- The impact of FoM on students in the five districts;
- The impact of FoM on its partners and on mathematicians;
- The institutionalization of FoM approaches and activities.
In addition, PERG evaluators carried out case study research on 6 teachers who participated in the MMT program.
Reports: (can attach to email for upload)
Lead Evaluator(s): Sabra Lee and Carol Baldassari
Associates: Rosalie Torres
Partners: Boston University, EDC, teachers from Chelsea Public Schools, Lawrence Public Schools, Watertown Public Schools, Waltham Public Schools
Funder: National Science Foundation – Math Science Partnership Program, supplemental award
Studies of four middle and high school mathematics teachers that completed Boston University’s Masters of Mathematics for Teaching Program were conducted between 2005 and 2008 to examine whether and how the program contributed to teachers’ deepening their knowledge of mathematics, shifting their instructional practices, and assuming leadership roles in their schools and districts.
Data collection included observations of FoM professional development sessions including those led by the case study teachers; multiple interviews with case study teachers, interviews and surveys at different points in time with FoM program PIs and staff members, Boston University faculty [mathematicians and mathematics educators], district representatives, and the four mathematics teaching fellows participating in the study; classroom observations; and program artifacts.
Reports on the research methodology and one case study have been completed. The writing of the second case study is in progress.
Attached: methodology developed for case study research
Lead Evaluator(s): Carol Baldassari
Associates: Elizabeth Osche
Partners: Gulf of Maine Research Institute
Years: Front-end evaluation: 2004-05
Summative evaluation: 2008-09
LabVenture! at GMRI’s Cohen Center for Interactive Learning
LabVenture! opened its Mystery of the X-Fish in fall 2005 and since then, has invited 5th and 6th grade students throughout Maine to visit the Center free of charge. The ‘mystery’ is based on a keystone species in both the Gulf of Maine foodweb and the state’s coastal economy – the Atlantic Herring. The visiting middle school students use a range of scientific methods and tools to study current, locally relevant research questions. Its ‘hands-on, minds-on’ activities are modeled on actual scientific research being conducted by marine scientists at work along the coast of Maine.
The Center’s LabVenture! program was designed to:
- stimulate student learning about marine science and the Gulf of Maine community
- illuminate current research projects and keep pace with scientific discoveries
- personalize the scientific process
- inspire and enable visitors’ ongoing learning
- generate excitement about science-related careers
After visiting the Center, students have access to personalized websites, with digital scrapbooks of their experiences (microscope images, team photos, video research reports, etc.). Students can also use the site’s online opportunities for self-directed learning and have an active dialogue with GMRI’s research staff. To support teachers, the website provides pre- and post-visit classroom activities.
PERG conducted the front-end and the summative evaluation for the Center. The purpose of the summative evaluation, conducted at the end of 5 years of operation, was to learn the extent to which, and how, teacher and student visits to the Cohen Center influenced learning and teaching both as a result of the Center’s activities at the time of the visit; and post visit. Areas of inquiry included:
- documentation of students’ and teachers’ experiences at the Center, as well as resultant follow-up activities
- evidence that visits had an impact on classroom teaching and learning of science, as well as the science curriculum
- evidence of teachers’ and students’ effort to extend their learning post-visit
- students’ and teachers’ plans for further research in marine sciences
Data collection methods included:
- on-line surveys of all teachers that visited the Center since it opened, as well as a sample of recent student visitors
- site visits to two schools; interviews with teachers and focus group interviews with students who worked together in teams during their visits
- review of student and teacher artifacts
GMRI: Vital Signs
GMRI: VitalVenture: NOAA B-WET program
Lead Evaluator: Judah Leblang
Associates: Samara Hoyer-Winfield, Elsa Bailey, Toby Atlas
Funder: National Institute of Health (NIH)
Abstract: The Infectious Diseases exhibit, and its related component on the Koshland Museum website, were funded through a Science Education Partnership Award (SEPA) grant from the National Center for Research Resources, a department of the National Institute of Health (NIH). The primary goals of Infectious Diseases were to:
- Help visitors understand why infectious diseases continue to cause problems and cause challenges for mankind—including the rapid evolution of microorganisms, the overuse/misuse of antibiotics, and issues of land use and economics;
- Help visitors understand what actions they can take to help control the spread of infectious disease;
- Pique visitors’ interest in the topic of infectious disease and stimulate questions.
As outlined in project documents, Infectious Diseases was “intended for non scientists adults in the general public, teachers, school groups at the middle school level and above, journalists, and policy makers.”
Evaluation Activities and methods:
PERG’s evaluation consisted of several phases.
- A series of front-end interviews with visitors to determine their understanding of and interest in the topic of infectious disease
- Exit interviews and interactive observations and follow up phone interviews with visitors at the Koshland Museum
- A series of interviews and a focus group with users, (including general visitors and educators) of the Infectious Disease portion of the Koshland website
- A post-summative-report round of interviews and observations at the Koshland Museum to examine visitors’ reactions to changes/adjustments made to the exhibit
Lead Evaluator(s): Judah Leblang
Associates: Samara Hoyer-Winfield
Partners: ITA (Idara-e-Taleem-o-Aagahi, the cooperating agency in Pakistan)
Funder: U.S. Department of State, Bureau of Educational and Cultural Affairs.
The Pakistani Educational and Leadership Institute is funded through a grant provided by the US Department of State, Bureau of Educational and Cultural Affairs, to Plymouth State University in New Hampshire. The project has been funded annually since the 2003-04 school year. A total of approximately 100 individuals have completed the institute over the past five years.
Each year the focus of the institute changes, to meet participants’ needs. The 2008 institute focused on the areas of cultural heritage preservation and environmental stewardship.
The institute is designed to:
- Contribute to the development of the Pakistani educational system (with a primary emphasis on government-run schools), by exposing teacher-trainers, educational administrators and other leaders to new educational methods and ideas.
- The institute also encourages cross-cultural communication and understanding by bringing Pakistanis and Americans together, or as stated in project documents: “to promote goodwill and understanding between the two countries.”
Data collection methods included:
- Interviews with attendees of the PELI institute
- Interviews with the program coordinator at ITA
- Ongoing discussions with the PI/program director at PSU
- Interviews with faculty and staff members
- Examine the PELI blog, surveys and other project artifacts
- Observations of several sessions of the institute
New Haven, CT
Lead Evaluator: Judah Leblang
Associates: Elizabeth Osche
Funder: National Science Foundation (NSF), Institute of Museum and Library Services (IMLS)
Abstract: The EVOLUTIONS (Evoking Learning and Understanding Through Investigations of the natural sciences) after school program began in 2005 at the Peabody Museum of Natural History at Yale University, and initially involved approximately 40 students. The program was initially targeted toward low-SES/minority students in the New Haven Public Schools. During the first year, the program targeted students in grades 7-12. Currently the program is aimed at high school students.
Annual projects include the mounting of an exhibition and student-produced videos intended to teach science concepts to elementary-aged students. Students also have the opportunity to participate in field trips to regional science centers in addition to an annual college visitation trip.
EVOLUTIONS program foci include:
- Preparing students for post-secondary (college) education;
- Learning about scientific—and other careers;
- Expanding students’ transferable skills for the future;
- Learning about the Peabody Museum and museum careers
In 2008, Peabody staff developed a new program, SCI CORPS, (Science Career Orientation and Readiness Program for Students), which enabled EVOLUTIONS veterans to train to work as interpreters within the Peabody Museum, and to interact with a broad range of museum visitors.
Evaluation Activities and methods included the following:
- Initial strategic planning—serving as a ‘critical friend’ to project staff and development of logic model for EVOLUTIONS
- Ongoing consultation with project director
- Observation of program through two annual site visits
- Numerous focus groups with students in both EVOLUTIONS and SCI CORPS
- Interviews with project director and staff
- Examination of project documents and artifacts
- Interviews with parents of participating students