Thursday, September 2, 2010

Crappy! Chart! Thursday (ish)! Fun with Performance Management History!

Seattle Public Schools is, we're told, All About Accountability. The daily reality is... hmmm. Much different. Still. Let's pretend we've got accountability. While we're at it, I'm going to pretend I can afford the you-must-be-joking level expensive but extremely fun McQueen peep toes with bedazzled skull decorations. Because ridiculous shoes make me happy. Sigh. Where were we? Right. Accountability.

The Performance Management Framework is part of The Strategic Plan. It's intended to create additional transparency, further its goal of everyone being accountable, and basically... make stuff better. Individual schools and the district as a whole are supposed to have annual, clear evaluations in the form of "report cards" which will allow everyone to see how they're doing. High-performing schools making progress with students can earn additional autonomy, and we can all have a look at what they're doing that helps their students succeed. Schools with poor performance and struggling student progress will receive additional assistance and supervision from the central office. The MAP test would allow everyone to gauge how efforts are working, and make adjustments accordingly, at multiple points during the year. For all schools participating, there would be additional funding based on their performance needs.

This all sounds fine in theory. Don't you want struggling schools to get additional help so their students do better? Shouldn't schools that are doing well by their students get some additional decision-making freedom? Do you hate babies, puppies, mothers and apple pie? Hint: You should only be answering no to the last question.

The idea behind Performance Management makes sense. The transparency it would give to everyone - to staff at schools, to the central office, to the community - would be a good thing. The increased ability to compare what efforts do and don't work and what sticking points might be could be great. And for those of us who only clean our proverbial houses (ahem. Not that I know anything about that in the literal sense) when we think someone might be coming over, it would hopefully promote increased accountability.

The SE Initiative, a district intervention in 3 schools (Aki Kurose, Rainier Beach HS and Cleveland HS), was the initial pilot for Performance Management.

All three schools have struggled with academics. Aki and Rainier Beach have been bleeding enrollment over the course of the last decade. The basic goal of the SE Initiative was to make things better at all three schools. Yes, the official wording was significantly more pompous than "make things better." The district explicitly stated that the SE Initiative would "inform the development of a comprehensive school performance framework." (p11, and yes, a long way of saying "it's a pilot")

So, did the SE Initiative work? On numerous fronts, no.

Enrollment is up a little bit at Aki and Rainier Beach. So that worked... a little. Even so, Aki and Rainier Beach are both woefully under capacity. I will briefly note that the glittery magic of functional capacity analysis reduced the capacity of under-enrolled schools like Aki and Rainier Beach, but conveeeeeeniently found that over-crowded schools like View Ridge and Schmitz Park had puh-lenty of room. Right. Anyway.

Academically? The program failed. Academics went up and down a little bit, but never hit the "met goal" mark (which meant coming within 10 points of the goal).

And transparency/accountability? Failed. The best document that I have reviewing the overall performance of the SE Initiative is one I obtained through super-sleuthing (okay. Someone sent it to me). And it pretty much... sucks. Take a look at it. Three years, and the best analysis they can provide is "progress made (y/n)" and "goal achieved (y/n)"?

No line item for total annual project cost, much less a line item breakdown of how the money was spent at each school, or whether the spending moved the student achievement needle. No note of tactics. Just whether there was progress or if the target was met. And I'll say again: the target was considered "met" if they came within 10 points of it. That's much like saying I got an "A+" because I came within 10 points of it. Which would be a... "B+". Still. Whatever. Those are the metrics they settled on. Maybe that's okay.

How-ev-uh. Given the significant lack of relevant information, at some point, you'd think someone would say, "the analysis of this pilot sucks." And since the SE Initiative was intended to "inform" a larger, rigorous performance framework, who in hell would say "Looks great! Let's... expand this!" ?

Unfortunately, SPS's board and management.

In 2009-10, Performance Management expanded to add 9 more schools, bringing the total number of schools piloting the "framework" to 12. They were:

1. Dearborn Park Elementary

2. Maple Elementary

3. Olympic Hills Elementary

4. Roxhill Elementary

5. Schmitz Park Elementary

6. Stevens Elementary

7. Aki Kurose Middle School

8. McClure Middle School

9. Mercer Middle School

10. Ballard High School

11. Cleveland High School

12. Rainier Beach High School

Results? Dunno.

Spending? Dunno. Well, publicly, dunno. The FAQ helpfully says that the Alliance for Education helped "support" the effort of creating the scorecards (dudes. For realsies? You needed a freaking grant to think up the scorecards?) with a grant of... oh, wait. They don't say how much it costs. However, a handy-dandy public records request indicated that the 2009-10 spending for Performance Management was $2.8 millllllllion ($1.5m from baseline and $1.3m from the Gates Foundation... which must have been passed through the Alliance for Education and then on to the district? Maybe. I dunno.)

Overall Goals? Dunno. Well... presumably to make things better for students? Let's go with that. We'll call "Make Stuff Better for Kids" a goal. In district-speak it would probably be more like "delivering on our commitment of ensuring a high quality school for every student so that all students graduate from high school prepared for College!, Career! and Life!"

Tactics Implemented to Achieve Goals? Dunno.

Any Tactics Changed to Optimize Results? Dunno.

To be fair(ish), it's possible that all of this will be reported on in detail in the Quarterly Strategic Plan Update. It's also possible that I will get my very own fairy godmother who will put the bedazzled-skull McQueens under my pillow the very same night that rigorous detail on Performance Management progress to date is publicly given. I don't know with absolute certainty whether either of this things will or won't happen. If you don't like obscenity, skip to the next paragraph, because I can't say this without swearing: I can sure as fuck guess what's likely, and sadly, it doesn't include a pair of kick-ass McQueens under my goddamn pillow.

So what do I know? I know that I can provide metrics (and a crappy chart!) for what I don't know, and then measure and benchmark the lack of information across multiple areas. Wooooooooo! Accountability, here we come!

Given the sketchy details and assessment for the SE Initiative, which involved 3 schools instead of 12, I'm not going to hold my breath that we're going to get more detail than I've provided in the above chart.

So. There seems to be a leeeeeeettle misunderstanding, by SPS management, of what a "pilot" is. In most places, a pilot is a small-scale project used to test the design/feasibility of a larger-scale project. Design issues in the pilot are often remedied if the project is moved forward to a larger scale. Often, a pilot provides quantitative proof of the viability of moving forward on a larger scale. This is how most people and groups think of pilot projects.

For SPS? We just keep expanding, baby. Changing course because of "data" and "results" is for sissies. And speaking of expanding...

For 2010-11, Performance Management will be rolled out in an additional 25 schools, bringing the total schools participating to 37. $5.1 milllllllllllion additional dollars will be allocated to participating schools, with grants ranging from $440,250 (Cleveland, with a 2009-10 FRL of 66.8%) to $24,000 (Brighton, 2009-10 FRL of 80.1%).

Given the financial shortfall in the operating budget, and the belt-tightening going on in schools across the district, you kinda have to ask: where's this five point two milllllllion dollars coming from?

As it happens, I know the answer to that, from page 10 of this presentation. It comes from the following funds: Title I ($2m), LAP ($.5m) and FRL ($2.1m).

But, kids, that's going have to be for the Crappy! Chart! Thursday! Fun with Performance Management History: The Sequel! That's right. There's (maybe. If I feel like it) going to be the first ever Crappy! Chart! Sequel!

It's historic. It's thrilling. It's sunny out.



Steve said...

Thank you. This is great info. Question: can the District simply pull from Title 1, LAP and FRL without approval from the board (the only people who are supposedly accountable to parents)? Are those funding sources supposed to be used for such things?

Charlie Mas said...


You keep using that word. I don't think it means what you think it means.

I know what I mean when I say accountability. You may know what you mean when you say accountability. But - and this is the $64,000 question - what does Dr. Maria Goodloe-Johnson mean when she says accountability?

From the Seattle Times
on Monday, Oct 15, 2007 at 12:52PM:
"Accountability means that Seattle Public Schools understands our data and we use it to set performance targets for the district, school and classrooms. We decide what our data indicators should be, i.e.: WASL, dropout rates, teacher retention, etc. A concrete example about accountability is SPS will decrease the achievement gap in specific sub-categories of students. The target should be reasonable based on the current data. Then the strategies should align toward making a positive impact. Evaluation is tied to ongoing assessment and holding all levels of staff accountable."

Ah ha! So, for the Southeast Education Initiative, did we have reasonable targets based on current data? Let's say that we did. The targets were secret, but we now have learned that there were some. Did the strategies align towards making a positive impact. Clearly, they did not. Was evaluation tied to ongoing assessment? It was not. Was staff held accountable? They were not.

Our model for accountability is one of no accountability.

Meg said...

Steve - I think that because the policy of Performance Management was approved, it gives district staff some leeway in creating funds for projects related to it.

However, regarding Title and possible also LAP, I think the district may be in violation of the guiding rules related to the funds ("supplement, not supplant" being the big one that SPS may be in violation of).

Charlie - I always love a good Inigo Montoya quote. In this case, since I'm not going with "inconceivable!" (which would be a good word to apply to some of the Superintendent's violations - since she was personally called out on the accountability audit for 3 different things, one of them being an ethics/conflict of interest violation for her presence on the NWEA board) I'm pretty sure the word means what I think it means. I suspect, though, that the central office feels that accountability is for everyone else.

Anonymous said...

I'm working backwords on this blog...So, is the Don Cherry jacket somewhere in there in the details, or does it require filing a Freedom of Information request and waiting the subsequent 10 weeks for a response from the District?

Maybe, Uncle Steve can put a positive spin on this, like he did when he said the SEA members voted "NO-Confidence" after they voted to ratify the CBA because they were "confused" (!!??)
ken berry

Dorothy Neville said...

So here's my question. IF our Initiative or Framework or whatever is only concerning Title I or LAP eligible schools, then fine and dandy for the district to selectively withhold some of that money and use if for said Initiative. It means that eligible kids get unequal access to the money, but NBD. It's allowed.

However, once the district makes the Framework apply to the whole district, then they are in hot water? Because then, they can't say that they have this cool program for all schools to benefit. Sure, we are using Title I and LAP money, but we promise that that money will only pay for Framework stuff in eligible schools. fer shure. That, to me, sure looks like supplanting instead of supplementing.

And what happens when Cleveland all of a sudden gets a huge influx of middle class kids instead of LAP eligible kids? Are future LAP dollars earmarked for STEM?