I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Sunday, March 23, 2014

HoLa Hailers Have a Hard Haul Hatin' on Hoboken High

UPDATE BELOW

Not all charter schools are havens for "white flight." But the ones that are have honed their defensive arguments down to this:
  1. We're awesome!
  2. We educate the same kids as the public district where we get our funds (even if we don't).
  3. The local public schools suck. And the comparison to us is valid, because we educate the same kids as the public district where we get our funds (even if we don't).
  4. Did I mention we're awesome?
I really have no problem with #1 and #4: who isn't proud of their kids' school? But #2 and #3 are easily verifiable -- and, in my experience, it turns out in most cases the claims just aren't true.

For example: let's head over to Hoboken -- the charter school segregation capital of New Jersey -- and see what one of the more academically credentialed charter cheerleaders there has to say about the local high school:
It has become a popular strategy to have the public expect less from schools and districts that educate students from families with lower than average socio-economic status (SES). Indeed, there is research and data that support the fact the there is a high correlation between SES and national tests like the SATRecently, a report was released that outlined very specifically the correlation of SES and the SAT. And, as one might expect, as family income increases so too does SAT scores (reading, math, writing, total or composite-see diagram). Using these numbers and charts, we can find an "expected" SAT score for each of 10 socio-economic categories. If a district's scores are higher than expected, we can reasonably assume something good is going on. Students are performing at levels higher than expected by socio-economic status alone. Conversely, if a district's SAT scores are lower than expected by socioeconomic status, we can reasonably assume something(s) is flawed educationally within the district.

As an example, let us look at the Hoboken School District in Hoboken, New Jersey. According to latest figures, the free lunch percentage (as opposed to "free and reduced percentage") for the Hoboken School District is 47% (Researcher Bruce Baker has said that "free lunch" is a better indictor of poverty on academic performance than "free and reduced"). While 47% is certainly challenging, it is not the lowest in Hudson County. Here are some other numbers for comparison: Bayonne: 52%; East Newark: 76%; Guttenberg: 71%; Harrison: 63%; Hoboken: 47%; Jersey City: 67%; Kearney: 23%; North Bergen: 55%; Secaucus: 19%; Union City: 84%; Weehawken: 46%; West New York: 72%

If we look at the LOWEST income category ($0- $20,000) we notice that we would expect a school district with students from this category to receive a Total SAT score of 1326. If we look at the second lowest income category ($20,000-$40,000) we would expect students from this demographic to receive a Total SAT score of 1402. Hoboken scores a SAT Total score of 1192. The majority of students in the Hoboken School District who take the SAT certainly fall somewhere in the lowest and second lowest categories. So what do we find?

Finding 1 (red line): Hoboken scores on the composite SAT are lower than would be expected by students from families making between $0-$20,000 a year. Moreover, this difference is VERY STATISTICALLY SIGNIFICANT (not occurring by chance).

Finding 1 (blue line): Hoboken scores on the composite SAT are lower than would be expected by students from families making between $20,000-$40,000 a year. Moreover, this difference is EXTREMELY STATISTICALLY SIGNIFICANT (not occurring by chance).

This analysis brings objectivity to the discussion of quality of education, especially in the Hoboken Public Schools. While its easy to have articulate, well intentioned parents to speak of the "wonderful" experiences their children are having in the schools and to hear the "good news" that often takes place at Board meetings, this analysis examines objective data in an unemotional context.

What does this mean? In common language it means that students attending the Hoboken Public Schools are performing at levels MUCH LOWER than would be expected based on their socio-economic status. 

Yeah, uh...

No. Even when using red fonts -- no.

Listen, I enjoy a good (?) t-test as the next guy, but let's be clear: you can't compare the average outcomes for a single high school against the average outcomes for an entire national income group. It's false and it's wrong and it's irrelevant and it's wrong and just no and wrong and don't do it, please.

We just don't know how Hoboken High's -- or any school's -- income is distributed through its population of kids who take the SAT (which is a voluntary test). If you want to see how a particular high school does against other high schools -- when controlling for student population socio-economic status -- then that's what you need to look at. Compare a high school against others by some metric that can control for all of their students' poverty status -- that makes sense.

But saying a high school doesn't do as well as a particular family income group of students nationwide is like saying your lawnmower sucks because it doesn't go as fast as your bicycle. Why would you ever think to compare the two?

There's also the little problem of using the SAT to judge a high school's performance -- something it was never designed to do -- but let's set that aside for the moment...

Instead of comparing Hoboken High to an entire class of students across the nation at a particular family income level, why don't we look at every high school in New Jersey -- including Hoboken High -- and judge their SAT total scores against the proportion if kids they have in poverty. That might make more sense, dontcha you think?

Me too:



I'm sorry - what was that you said again, professor?
If a district's scores are higher than expected, we can reasonably assume something good is going on.
Yeah, OK, sure: after all, the correlation between SAT scores and percentage of free lunch students is very tight. Nearly 70 percent of the difference between schools in their SAT scores can be explained by those schools' Free Lunch-eligible percentage (look at the r-squared).

The trendline shows the prediction of where a school should be when accounting for free lunch eligibility among the student populations. Hoboken High is actually above prediction: Hoboken High is above where we would expect to find it on SAT scores, even when controlling for student poverty.

Must be "something good is going on," right?

Folks, we can and should have an honest conversation about education and poverty and urban demographics and the role of charter schools and testing. Let's have that.

But beating up on your local high school with dubious comparisons? Yeah, not really helping...


ADDING: Darcie points out that Anthony Petrosino, the author of the post above, is a trustee for HoLa.

ADDING MORE: One of the most disappointing aspects of the Hoboken charter wars is how easily bad data has spread on this issue. Case in point:

Petrosino gives these figures for Hudson County school districts' percentages of free lunch eligible students:
While 47% [Hoboken] is certainly challenging, it is not the lowest in Hudson County. Here are some other numbers for comparison: Bayonne: 52%; East Newark: 76%; Guttenberg: 71%; Harrison: 63%; Hoboken: 47%; Jersey City: 67%; Kearney: 23%; North Bergen: 55%; Secaucus: 19%; Union City: 84%; Weehawken: 46%; West New York: 72%
Except these figures are NOT the FL percentages for the school districts. How do I know? I wrote the brief Petrosino cites.

These figures are what I termed in the brief the "actual" FL% for the district: the FL% when including any of the charter schools within the district's borders. I did this to calculate a "disparity ratio" for schools in Hoboken and Jersey City, showing how much they differed from their surrounding district. Charters, unsurprisingly, did very poorly on this measure -- they just don't serve the same percentage of children in economic disadvantage as their public school neighbors. HoLa is particularly bad.

Look, I make mistakes all the time, so I'm not going to bust on anyone for misreading my brief. But if you're going to use data to lash out at a school that serves the economically deprived children your school does not serve...

Well, maybe you'd better double check your "facts" before you start taking shots:
Recently, Hoboken Superintendent Toback has discussed the so called "segregating" impact of charter schools on the school district. Board President Leon Gold has been quoted discussing the "white flight" occurring in the Hoboken School District because of charter schools. Perhaps these two gentlemen should concentrate on providing a better educational experience for the students already attending their schools and be less concerned with baseless statements of segregation, "white flight",  and a climate of lowered expectations for the students and families they are already underserving. -Dr. Petrosino
I'd say Dr. Petrosino owes both Toback and Gold an apology.

12 comments:

Frankie Adao said...

Thanks...another great article. Informative! Data Driven.

P. Grunther said...

Thanks JJ. It is so important to have people deconstruct the BS when these misleading articles are published. The average reader has no idea how statistically significant statistics are arrived at, what a T-test is, etc., etc. The problem, of course, is that we have absolutely no guarantee that the people who read these types of articles have access to or even care to read blogs like the present one which serve to counter the BS that the public is being bombarded with...alas.

Giuseppe said...

It is appalling that the pro charter mafia get away with all their propaganda. Thanks to JJ, Diane Ravitch, Bob Braun, Mother Crusader and many others for debunking the mythology and spreading the truth.
Side note: There was a tragic and terrible school fire in Edison, the school was completely destroyed. No one was in the school, no one killed or seriously injured, the good news. But Edison schools are so overcrowded they have no idea where they will put the 500 kids who attended that school. Will Christie step in and use the disaster to fire all the teachers and set up 4 charter schools to accommodate the kids from the "failing school?"

jcg said...
This comment has been removed by the author.
jcg said...

These edu-bidness, jingoistic views of education are warped. We'll become a third world country if we continue believing charter-cheerleaders fairy tales.

Dr. Anthony Petrosino said...

I have included Effect size and Cohen d for some additional information. If a school has a FREE LUNCH of 47% and a FREE AND REDUCED LUNCH of > 75%, we may not know the exact specifics of the population distribution but we can make some reasonable assumptions.

If more and specific data was available, I have no doubt a more detailed analysis could be conducted.

I believe comparing a high school and a national income group along the same measure (SAT composite) using a t-test, while not perfect, is reasonable. There are other ways of comparing population means.

I agree it would be informative to do such an analysis at the county, state or some other level as well. To that degree, your chart deepens the conversation and is appreciated. But the chart does fall outside the present analysis conducted.

Duke said...

Dr. Petrosino, we do, in fact, have "more and specific data."

The 2013 enrollment file shows Hoboken High has 592 students; 411 qualify for FL, which is about 69% (about 81% for FRPL). The district FL% is about 64%.

The 47% you cite is from a report I wrote. That is what I termed the "actual" district FL percentage: the overall FL percentage including the sending charter schools. Since you are using these figures to make the case that HPS is not doing a good job, you obviously can't use the children the district is not educating to calculate your FL percentage for HHS. Agreed?

So, no, we can't make the "reasonable assumption" you make that comparing a national income group (with no adjustment for regional differences, I will add) to HHS is apt. What we ought to do - what, as I showed, we actually CAN do - is compare HHS to other high schools and make some reasonable attempt to control for economic disadvantage.

It turns out that HHS is just about where we'd expect.

BTW: I have some serious doubts about your t-test that I didn't really get into. How do you know the variance of the 50 (!) SAT test takers at HHS matches the variance for the 300K in the national population? Do we have any idea how the scores are distributed for the 50 HHS test takers at all? Is it normal? If we don't know, why are we running a t-test?

You make an awfully strong assertion here:

"It is fair to say based on this data and analysis that students from the Hoboken School District are scoring at levels MUCH lower than expected based on their family income.

Recently, Hoboken Superintendent Toback has discussed the so called "segregating" impact of charter schools on the school district. Board President Leon Gold has been quoted discussing the "white flight" occurring in the Hoboken School District because of charter schools. Perhaps these two gentlemen should concentrate on providing a better educational experience for the students already attending their schools and be less concerned with baseless rantings of segregation, "white flight", and a climate of lowered expectations for the students and families they are already underserving. -Dr. Petrosino"


Doctor, if you are going to charge the HPS with "underserving" their students, I would suggest you'd better have your facts all in a row, ready for a challenge. I gave you such a challenge, and I believe your analysis can't withstand it.

The burden of proof, then, is on you. I have shown that HHS is, using your preferred metric, EXACTLY where we would expect it to be.

Why should we dismiss my evidence?

sockmonkey said...

Not really helping?!! He’s been beating the war drums for the Hoboken BoE since they didn't renew his contract. It was discovered Dr. Petrosino was working in Texas while being paid for a full time position as an Assistant to the Superintendent in Hoboken. Sweet, low-show job. Angry that he wasn’t renewed, he sued the district and lost. He actually had to pay some funds back because he simply wasn’t there. That dude has an ax to grind and his blog seems to be focused on bashing the Hoboken Public School district. He’s a board trustee for the HoLa Charter School in Hoboken.

LeAnn Lewis said...

That doesn't surprise me that he came to TX...all the grifters and carpetbaggers show up here. :(

Dr. Anthony Petrosino said...

I would like to acknowledge and thank a number of anonymous bloggers and colleagues who have provided useful data, readings, perspectives, url's, and analysis on this topic. Edits have been made to the original post on 3/23 and 3/24 and now include the use of Cohen's d, effect size, links to free lunch (FL) and free and reduced lunch (FRPL) criteria, deleting of county comparisons (the FL% when including any of the charter schools within the district's borders), explanation of some assumptions, an alternative one-tailed t test analysis, and modifications to some of the original text.

Duke said...

Dr. Petrosino:

1) I'm not anonymous, as the "About the Jazzman" page above makes clear.

2) You could put a dozen statistical tests up comparing HHS's average SAT scores against a particular income group's national average SAT scores, and it wouldn't matter a bit. The comparison is not apt. Regional differences in purchasing power alone would negate the comparison.

The better statistical tool is a linear regression using some proxy measure of economic difference against student outcomes, which I have done (admittedly, with the caveat that SAT scores are not great measures of school effects even when accounting for economic differences).

3) You used this comparison to take a swipe at HPS, its superintendent, and the president of its board. Aside from just being a plain old cheap shot, your analysis is flawed, which means you criticism is built on a foundation of sand.

Cohen's d doesn't change that reality one bit.

Dr. Anthony Petrosino said...

1) my statement was meant to cover a number of people who commented about my post-- including you but not exclusive to you. Do you prefer to be referenced as "Duke"? I did not find a name on your profile.
2) I agree there are a number of other measures to look at in terms of "school effects" and SAT score is certainly not the only measure available. There is graduation rate, state test scores, violence and vandalism reports, as well as rankings by independent news organizations. I do not disagree that linear regression using NJ high schools adds to the conversation.
Your multiple regression model does not paint a particularly good picture either. As I read it, the HHS composite SAT scores are close to what would be expected based on income levels of a statewide cohort (slightly above, I am assuming not statistically significant by eyeballing your chart). What then is the added value of the formal educational experience? Even accounting for the typically accepted "70% of variance/SES"-- what impact is the other 30% having on this measure?
3) At the core we have the question, are the means different? And if the means are different, are they different by more than we would expect simply by chance. I believe the "sandy" foundation is firmer now thanks to comments you (and others) have made and I tried to acknowledge that in my revised statement.
IMHO, there is nothing inconsistent with HHS being close to expected at the state level (as you show with your linear regression model using NJ HS data) and lower than expected at the national level using an analysis of means. I am in the process of trying to find the SAT composite averages binned by income level by state but have had no success to date.
Again, thank you for the comments and the interaction. Will you be at AERA next week?