Sunday, December 5, 2010

Crappy! Chart! Thursday! Simple Cons: Beating the Last 17% Out of a Dead Horse

I wasn’t going to post about 17% Miscommunication-Gate, which has mostly been beaten to death while I’ve been holed up with antibiotics, cough medicine, Tylenol, and lots of tea. There's probably not a ton more to say about this.

But sometimes it’s fun to beat a dead horse. So, here goes.

District Statement

This is long and Charlie Brown grown-up wah-wah-wah-wah, even excerpted, so bear with me. And try to stay awake.

The Superintendent sent this letter to various members of the “Seattle community.”

“In 2008, Seattle Public Schools (SPS) published a conservative data point aimed at determining the percent of students that graduate from SPS ready for a 4-year college. This specific data point is complex and one that districts across the state and the country grapple with as they try to quantify college and career ready. College readiness measures can be defined in multiple ways: the minimum requirements necessary to graduate high school, minimum requirements necessary to apply to a 4-year college, minimum requirements to successfully enroll in a college or university or meeting the necessary requirements to succeed in and graduate from college.

At the time we calculated that 17% of our students graduated from SPS college and career ready, we used a very aggressive standard to determine the percent of SPS students that were college ready based on our understanding of what is needed to be admitted and succeed in college, not simply the minimum requirements to apply.

This specific data point sparked significant public dialogue. In 2009, we chose not to include this statistic in the initial release of the district scorecard because we wanted to review it further; we publicly announced it was under review. In 2010, after additional research and discussion, we revised the statistic on the district scorecard using reduced math and science requirements as well as a reduction in the minimum core GPA from a letter grade of “B” to a “C” that are more in line with the Washington Higher Education Coordinating Board (HECB) minimum requirements to apply to college. Further, at the 11/17/2010 board workshop, the district stated that the statistic changed and provided an explanation.

In retrospect, this review should have been accelerated and we should have been more proactive, both internally with staff and externally with key stakeholders, when the original statistic was held back in 2009 and was under further review. In addition, we should have been clearer that this represented a standard more rigorous than the minimum HECB requirements.”

Actually, I can’t go past 4 paragraphs. Just reading it this far has me trying to stab a pen through my hand in order to stay awake.

You wanna to read the whole thing? Good luck. There's just a leeeeetle more bureaucratese to slog through. It’s available here.

Actual Factual Facts!

Factual! SPS did publish a data point in 2008! And it did spark significant public... something. "Dialogue" isn't the right word since that implies that SPS was actively engaging in back and forth with the public. Which, um, SPS is not exactly renowned for.

Not Factual! Pretty much everything else in the letter.

The data point in question is “Graduates meeting high school credit requirements for 4-year college,” [emphasis mine] and was one of 16 highlighted goals in the Strategic Plan.

The Superintendent spent her first year on the job authoring this plan.

Despite all that work on the plan, somehow a different metric, "Graduates prepared for 4-year college," is what was "assessed" and now appears in the district scorecard. Obviously, something changed.

At the June 4, 2008 board meeting (tune in at about 1:18:30), when the school board approved the Strategic Plan, Michael DeBell asked about this precise data point. He said that he felt 17% was too low.

The answer he received was that SPS used the Higher Education Coordinating Board's (HECB) definition of minimum college admission standards in order to determine how many SPS graduates met high school credit requirements for 4-year college. HECB's standards had been revised for 2012; the claim was that SPS used HECB's 2012 definition.

Curiously, nowhere during this explanation did the Superintendent pause to note that SPS wasn't actually using the 2012 HECB standards.

Today’s Crappy! Chart! is just a table that walks through the differences between those HECB standards and what the Strategic Plan presented as HECB standards. However, despite the genuine crappiness of the chart, I certify that it meets the minimum Crappy! Chart! requirement standards, as measured by me after a snork of cough syrup or glass of wine, whichever is handiest at the time of standards determination (Ha! That wasn't bad bureaucratese!). SPS added a science course and bumped the GPA up a point. Those two alterations were enough to knock a startling number of kids out of the running (29% of SPS high school graduates, if I recall correctly).

Truthy SuperAnalysis!™

Dudes. Totally gnarly epic fail.

The percent of SPS graduates meeting credit requirements for college was supposedly calculated by HECB standards, but... the district didn't use HECB's standards.

The Superintendent’s letter completely flubs effective minimization of the problem. I’m a little looped on my cocktail of medicine, cough syrup and Tylenol and I still don’t buy what the Superintendent’s selling in her letter. That the letter is in such sleep-inducing bureaucratese doesn't help.

I get that getting busted is no fun at all, and the forced explanations and apologies that go hand-in-hand with getting busted are deeply tiresome to make. Still, when you have to sell that an enormous misrepresentation was actually just a wee "miscommunication," you have got to work the hell out of that explanation.

There was ample time to correct members of the public who were misled by the deeply misleading data and came to the conclusion that by saying "requirements," the authors of the Strategic Plan meant... "requirements."

There were direct questions on the data, and in answering those questions, the Strategic Plan authors represented their 17% figure as conforming to a standard. Which... it didn't.

So why didn't the district calculate the actual percent of SPS graduates meeting credit requirements for college using the 2012 HECB standards (as they said they had)?

There's the conspiracy theory guess: management didn't want to do any work to improve that particular stat and hoped, in 2013, to slide the real number in and then voilà! they'd have met the goal without doing any work. That seems like a bit of a stretch, even to someone as suspicious as me. It looks like there was either intent or incompetence, although a combination is a distinct possibility.

In the absence of an honest explanation from the district, we can only guess.

Truthy SuperAnalysis!™ says: Whether by intent, bumbling incompetence or both, the Strategic Plan authors misrepresented how many SPS graduates were meeting credit requirements for college, misleading the board and public.

And Abelard says:

Are you sure you don't have some raisins?


Dorothy Neville said...

According to This HECB report the 2012 math requirements are still 3 credits. The only change is that one must take a math course as a senior. (OK, This was a revision/clarification published in 2009. So a time when SPS could have saved face and revised the metric, possibly hiding their con.)

The version of the letter I saw explained that the "aggressive" standard included "earning a letter grade of “B” or higher in each of their core classes." That's not the same thing as GPA. Was GPA used or was this letter so hastily put together no one read it over for obvious flaws?

Also, in the interview with Linda Shaw, Brad admitted that he undercounted in another way. He failed to include some CTE courses that should have been counted for math credit. He also failed to include courses taken outside of SPS, such as UW Summer Stretch. While SPS doesn't give credit for them, colleges do.

Scrawny Kayaker said...

Oh, please! *Real* bureaucratese would never use a word as colloquial as "handiest." You have to find the adjectivalizationalism of something noble-sounding like optimum, maximum, or harium.

But seriously: you rock *so* hard. You totally made that dead horse your female canid. Please don't stop posting!

Also, love your goats. I'm amazed they don't eat their parkas. Are those woven of kevlar and titanium wire?

dan dempsey said...

The point remains that the Strategic Plan was written in 2008 and since the analysis was for kids graduating likely in 2007. The relevant statistic to put forth would have been the percentage of students meeting the minimum credit requirements to enter a four year college at the time of graduation.

Note on page 11 of the plan it states:
17% of students satisfied Minimum credit requirements to enter a four-year college.

It seems only reasonable to put forth a figure that actually applied to the students graduating. The students and their counselors were not trying to meet 2012 or 2013 standards for admission but rather 2007 standards.

This is just another example of the Superintendent failing to report the relevant data and preferring to submit something less relevant in a deceptive way. That has become her hallmark in Seattle. Her title of "Dr. No Confidence" is definitely merited by her actions and words.

She should be under investigation by the AG for multiple violations of state laws.

Seattle Parent said...

Actually, the 15 HECB minimum credits required for college admission to WA public schools has not changed from 2007-1012 standards. Also, the 2.0 minimum average GPA hasn't changed either. The tweeks have been in things like the science credits have to include labs, and the math-related class in senior year. But the 3 math credits (not 4) and 2 in science (not 3) have remained the same. Brad B & the SPS just plain got it wrong (see the SPS Srategic Plan's Appendix).

Check the original sources- on the HECB website as they have the historical comparisons.

Also, for fun, check out this early 2008 WA State Board of Ed report about HS credits and the effect on college readiness when schools require more credits to graduate (does not raise college readiness):


On page 8 is a listing also of the HECB requirements, adopted May, 2007. The interesting thing is that this state-wide study by BERC used the HECB HS graduation requirements as a standard to measure college readiness (48.5% statewide average). This was presented to the State Board of Ed, as well as the CORE 24 High School Committee, which Michael Tolley from SPS was a member. Six months later SPS chose a completely different standard to measure college readiness? You would think that Michael Tolley, SPS High School Director at the time, might think 17% was a bit odd?

Seattle Parent said...