Identifying Learning Gaps

One Schoolhouse's April 15, 2020, Academic Leaders Webinar focused on "learning gaps" that may develop this spring, as students work in distance learning environments for the first time. Tom Rochon, President, and Glenn Milewski, Chief Program Officer, from ERB discuss: resources available this spring in order to assess student learning and well-being, and a new tool that schools can use next fall to understand what gaps emerged and how to target skill development over the 2020-2021 school year.

---

 Brad Rathgeber, Head of School of One Schoolhouse: So we are thrilled today to be joined by Tom and Glenn from ERB (Educational Records Bureau) to explore this topic of understanding what learning gaps might be created as we go through this crazy spring. Tom and Glenn, welcome. Thank you so much for joining today.
 
Tom Rochon, President of ERB: Thanks, Brad, thanks so much for having us here. As you know, it's an unprecedented time with obviously, lots of unique challenges for schools right now. I know assessments are not necessarily on top of the list of things that everyone is worried about, yet it rises very quickly to near the top of the list for a number of reasons.
 
Brad: Absolutely it does. The first question I'd love to just know a little bit about is: what resources is ERB making available to educators and families right now to help them through the COVID-19 crisis?
 
Tom: We know that assessment is most valuable for innovating schools. It's a way to show that students are learning what they need to know to demonstrate the effectiveness of the educational program despite or because of the school's experimental stance or directions.
 
And right now, all schools are innovating schools. Everyone is experimenting. We're in unknown territory, so we know at ERB and we've been working really hard to support what has been an abrupt and unexpected transition to distance learning. So, in the immediate term, to address your question directly, a number of schools have taken us up on the offer of the free Writing Practice Program for the rest of the academic year. This is a self-guided online A.I. driven writing instructional program that students can follow and they get feedback through the A.I. algorithm.
 
Teachers can monitor how much work they're doing and how they're progressing and put their own stamp on it by interacting with students.
 
Brad: For those unfamiliar with that program, what are the appropriate grade levels for the WPP? 
 
Glenn Milewski, Chief Program Officer at ERB: Typically the most popular grade levels for that program are grades 3 through 8. What the people who use the program tend to really value is the fact that the results are presented immediately after students submit their writing, and they're fairly detailed. There is feedback on six different traits and each trait has a particular rubric to guide the feedback that's given and so it does really help students practice and benefit from trying out different essay prompts online.
 
Tom: We also know that SEL work is one of the hardest to transition to a distance relationship. Yet at the same time, there's been just a glut of resources offered over the last month or so in that area. So, we are also curating a selection of SEL perspectives and exercises on our website.
 
And, then the third thing I'd want to mention in the short term is an announcement that the ISEE admissions tests will be available to students at home within a proctored and secure fashion so that schools can continue to do their admission testing through the spring and into the summer at the tail end of the full admissions cycle.
 
Brad: That's great. Can you talk through other ways that ERB is supporting schools with testing options? We know that a lot of schools are used to doing their CTP testing, for example, in the springtime.
 
Glenn: Tom mentioned the admission testing that we are doing with at home proctoring. We're also supporting achievement testing via the CTP with at home testing options. What's different about the two approaches is for the ISEE program, our partners at Prometric, will be proctoring the exam for students. So, schools won't be involved in that process and it will mirror pretty much the same procedures that students and families have when they enter a Prometric test center. Our members will receive test scores for the ISEE, but they won't be involved in administration. For achievement testing with the CTP, it's a little bit different. We're asking schools to administer the exam to their students at home and to follow the same procedures that we have in place for CTP online testing, which are well established and well owned, with the exception that teachers and administrators will be proctoring via a video conference using Zoom. We've got extensive materials to support implementation available on our website on the CTP portal.
 
We've heard from over 200 members that have interest in doing this and they really want to know what to plan for. One interesting point that came up today is that, we had our first school administer CTP online at home and several dozen students took the exam and they were able to obtain their scores immediately after the assessment. It was frankly a little bit bumpy, we sat in with the school and it was their first time doing CTP online testing. They were a paper school, and so they were kind of working out the kinks and this sort of new way, but I would count it as a success. Every time we administer another one of these at home, we're learning a little bit more and more adjusting directions for schools.
 
Brad: For schools that are used to running the CTP test in the spring. Would you encourage them to stay on that cycle right now and to try to move to the online format, or would you encourage them to do something different?
 
Glenn: The way that I always answer this question is to say that we want to support schools in what they feel they should do to best support learning. We’re trying to be as flexible as possible and give schools as many options as they need.
 
Tom: I think that perspective is exactly right, and there's obviously a sense in which this spring and summer will be unlike any other. So the data from whether a school tests at home with CTP this spring or moves to a fall testing cycle, there will always be a certain breakpoint or a certain asterisk next to it. We understand that, which is why we simply want to give schools as many options as they can have.
 
Glenn: And I want to actually build on what, if it's ok Brad, build on what Tom just said because ERB is launching a new program for back to school next year.
 
Brad: Let’s transition to that.
 
Glenn: ERB is launching the Milestones program to complement to the CTP. Except that it's a much lower intensity brief version of the CTP that only includes two sub tests, reading comprehension and math.
 
The reason I mentioned this in the context of COVID 19 is that we feel that for schools who simply cannot test this spring, they're not able to do at home CTP testing, the most helpful option for them would be to administer the Milestones assessment in the fall and to get a quick look at students overall level of learning and any areas in which they have particular strains or weaknesses. Many schools, I think are going to be feeling quite in the dark after a six-month hiatus of not having in-school learning. So what can really help them is a data point in the beginning of the year.
 
The way the Milestones program works as a complement to the CTP. Milestones is offered in three assessments in the fall, winter, or spring. And for schools that are administering the CTP in the spring, they can administer Milestones in the fall and winter. The first full assessment is this learning check. The second assessment in the winter is a kind of a follow up, so for students who performed particularly poorly or to just get a check on learning for all students mid-year, the Milestone assessment is available.
 
And then to your earlier question, schools that are typically administering the CTP in the spring, we recommend that they administer it again in the spring of 2021, and that will allow them to preserve their longitudinal trend data to have a very robust and fulsome look at all of the areas that we measure and a good tool for evaluating curriculum effectiveness.
 
Brad: So let me just recap on that to make sure that we get that answer super clear for schools. So schools have a couple of different options for the spring, they can move to online testing environment if they haven't moved to the online testing environment, and/or if they do not think that it's right for them to offer a spring CTP test date they can still get a snapshot of where their students are through the ERB Milestones program this fall and winter. It will help them again identify some of the gaps that may have been created over the previous six months.
 
I really appreciate and I think everybody on this call, appreciates the way that ERB is reaching out and helping schools both with free resources and thinking about things very flexibly in regards the mission program and the CTP program and the Milestones program probably couldn't have come along at a better time.
 
We do have a few questions that are coming in. Corbit is wondering, do you have any data around expected deltas when schools change? And I think that's from spring to fall testing.
 
Glenn: So one thing to keep in mind about the Milestones program and the CTP program is that we use a vertical scale so that accounts for different times of year when students will test. So, think of it like the scale in your bathroom. The measurements are always the same and what changes are the norms that we use to interpret those measurements. So, we have fall and spring norms for the CTP, we had those for a long time, but for the Milestones program will have fall, winter. and spring norms to help with those interpretations.
 
Tom: I really like to extend Glenn's metaphor, because on the one hand, if you shift from fall to spring or spring to fall, your school, your results would be compared against other schools that are testing in the fall. On the other hand, we all know that our weight fluctuates during the day and if we typically weigh ourselves in the morning and then we start weighing ourselves in the evening, we're going to get a different result. And while we might have a norm pool to expect, actually, I don't know what happens if we generally gain weight or lose weight during the day, we might know to adjust in that way, but you'll never be able to know exactly how your metabolism is, or let me drop this extended metaphor. Your school would actually be affected by that shift. So it does represent a shift that cannot just be statistically controlled or raised.
 
Brad: Another question that came in is will there be a skilled breakdown on the math? And I'm guessing as that's related to the Milestones program, too.
 
Glenn: Let me hit some of the high points in terms of the design. And by the way, for people who are interested in more information, we have a website specifically geared to this program. It's called It's at erblearn.org/milestones
 
Some of the high points when I mentioned that the Milestone program was a compliment to CTP, I failed to include the detail that it uses the same score scales and it will have the same content standards mastery scores as the CTP and it will also use the same norm pool. So basically, it's a short form of the CTP and that's what our members asked for. We conducted a lot of outreach with them and they said for this to be helpful from a pedagogical standpoint, we need to have results on the same scale and with the same level of detail and the same norms.
 
What we're also doing that's going to be different for Milestones and CTP as well is we're adding criterion referenced interpretations and that's really exciting. We have some technical advisors who said you're doing great on the norm reference interpretations, but it would be helpful to give some criterion reference feedback as well. So we're actually working with several dozen teachers to help us interpret what level of performance on these assessments reflects meeting expectations and exceeding expectations, so we'll be providing criterion reference feedback for milestones as well.
 
Brad: I want to make sure this question is answered fully: is there a way to compare the fall milestones data to the traditional end of year spring data?
 
Glenn: Absolutely, and it goes back to that scale metaphor, right? So imagine you weighed 100 pounds in the fall and you weigh 120 in the spring the plus 20 points is your growth over the year. And we can measure that reliably because we have these very strong vertical scales on CTP and Milestones.
 
Tom: And identify an expected growth from fall to spring so that you can not only understand the trajectories of the different students in a given class, but you can also understand what those trajectories look like against a general sort of external set of norms.
 
Brad: Are there other other places you can think of if you are a school leader that you would be really interested in digging into and identifying student gaps and learning as we get into the summer and fall?
 
Glenn: ERB has partnered with Re-think Ed to offer social and emotional learning programming and our contribution to that partnership has been to create an assessment of social and emotional learning competencies. It's very brief and it's really guided toward helping educators help students refine and build on their competencies. But one of things we realized very quickly after we started getting data back is what power there is in linking social emotional learning competency data with learning achievement data. And there are really important relationships that you can see in those data points. And moreover, there's this notion that if you're helping students develop in social emotional learning areas, you're also going to have a corresponding impact on their learning achievement, which we all kind of know intuitively, but seeing that effect in the data has just been very powerful for us in schools.
 
Tom: I'd piggyback onto that I mentioned at the beginning of the webinar that all schools are now experimental schools perforce by dint of the environment. And one of the aspects of that experiment, of course, is trying out different distance learning techniques. I'm sure that there's not a single uniform school philosophy that's managed to develop over just a few weeks. Different teachers are doing different things. And actually, Brad, I think it was from you that I heard the intriguing anecdotal finding that students who are extroverts might not be doing as well in the transition to distance learning because they are so relational, relationship oriented as students who are introverts. Yet another opportunity to understand the impact of this experimental moment and distance learning by combining data from our SEL measure in our CTP measure.
 
I guess what I would say in general is that the more data one can collect on student learning in this time and the more you can relate that to this to other aspects of how the school understand, how teachers understand, the students and their characteristics and how they learn, the better prepared the school will be to absorb the lessons from this moment and assuming that the future is going in a certain direction anyway. To use those lessons to become a really effective educational environment in what might be a future hybrid scenario of in-person and distance learning. This is a special moment and looking at it in a certain way, it's a glass half full moment where there's a lot of opportunity.
 
Brad: And so we should be thinking about measurement broader than we typically have in the past.
I think that's one of the key takeaways from this.
 
Glenn: I think for us too, unlocking the power of those measurements through better reporting platforms is really where our emphasis is. And again, I don't want this to be a pitch either, but we're really excited that we're making some important improvements to our reporting that will be a part of the Milestones program and CTP next year too.
 
Brad: Another question that we have come in is what would you recommend that we as academic leaders recommend to families to support their kids learning over the summer time?
Are there other tools, resources, places that from your perspective and where you said we should be thinking about offering out to families?
 
Tom: The summer melt phenomenon is long known and the COVID melt will presumably be even more severe because students haven't had the on-campus educational experience right up until June, so I think the question is very well-placed. I’ve mentioned the Writing Practice Program that we've made free for schools to use with students, and again a few schools have already taken us up on that. There's also a just a direct to family version of the Writing Practice Program that we have. Parents who are in direct contact with, because they've used an ISEE test in the past, we've made them aware of that. And quite a few parents have already. That's not free, that's something that parents have to purchase. But parents have already taken us up on that in fairly significant numbers. And so I think the philosophy of every parent is a little bit of a homeschool coordinator now. It's a philosophy that's being adopted.
 
Glenn: One other idea that I would mention as importance is project-based learning. A lot of the research is showing that that can have tremendous impacts on learning achievement. But it's also the sort of activity that a student can work on over an extended period of time, such as the spring and throughout the summer. On the other hand, I also know that parents, educators, and even the children are stressed right now. And what is also important for them to do is to kind of take a break over the summer as well and come back to school fresh and ready to learn.
 
Brad: Right. Well, again, Glenn and Tom, thank you so much for joining us today.