US News Law School Rankings -- 5 Failures Forum
- jenesaislaw
- Posts: 1005
- Joined: Mon May 19, 2008 6:35 pm
US News Law School Rankings -- 5 Failures
I just published a piece on Law.com entitled, Five Failures of the U.S. News Rankings.
Here is the body:
Tomorrow, the law school world will overreact to slightly-shuffled U.S. News rankings. Proud alumni and worried students will voice concerns. Provosts will threaten jobs. Prospective students will confuse the annual shuffle with genuine reputational change.
Law school administrators will react predictably. They’ll articulate methodological flaws and lament negative externalities, but will nevertheless commit to the rankings game through their statements and actions. Assuring stakeholders bearing pitchforks has become part of the job description.
If the rankings measured something useful, the entire charade would be much easier to stomach. The unfortunate irony is that these rankings adversely affect the decision-making process for law school administrators and prospective law students alike. The stakes are high. Our profession and society need law schools that don’t figure inefficient metrics into annual budgets. Dollars spent chasing U.S. News rankings diverts funds away from students’ education. It also stands in the way of reducing tuition.
In this post, I examine five U.S. News rankings failures. I consider the methodology and underlying rankings theory from the perspective of a student who features job prospects prominently in his application and enrollment decisions. Considering the near universal support for prioritizing job outcomes in the process, these failures demonstrate just some of the reasons annual consternation hardly seems worth it.
Word cloud of LSAT test-takers surveyed about the reasons they plan to obtain a J.D. degree. Source: Above the Law.
First, the rankings pay insufficient attention to what matters most to prospective students: job outcomes. In a survey of 600 students studying for the October 2012 LSAT, Breaking Media’s research arm found that the two most popular words associated with the students’ purpose of getting a J.D. were “career” and “work” (image above). These are not exactly shocking results. Despite the importance of job outcomes, they account for only 18% of the rank and credit schools for jobs few attend law school to pursue.
Second, the rankings use a national scope, which places schools on the same scale. Only a handful of schools have a truly national reach in job placement. The rest have a regional, in-state, or even just local reach. The relative positioning of California Western and West Virginia in the rankings is virtually meaningless. Graduates from these schools do not compete with one another.
Percentages in above graphic are of the entire class.
It turns out that 158 schools place at least half of their employed class of 2013 graduates in one state. The top state destination for each school accounts for 67% of employed graduates. A much smaller 8.2% of employed graduates go to a school’s second most popular destination, with just 4.5% of employed graduates working in the third most popular destination. Only 20.4% of employed graduates (16.7% of the entire class) end up in a state other than the top three. Comparing schools across the country just doesn’t make sense.
Third, U.S. News rankings follow an ordinal system that fails to show the degree of difference between schools. Are Columbia and NYU virtually tied? Or does the two-rank difference represent a wide gulf in quality? Is the so-called difference between Columbia and NYU the same as the difference between Cornell and Vanderbilt? Students weighing school prices need to know not just which school is better but how much better it is.
Fourth, performance changes over time but year-to-year comparisons are virtually impossible using the U.S. News rankings. U.S. News will tell you that Stanford knocked Harvard out of the #2 spot in 2012-13, but the swap in rankings does not provide any clues to your average reader as to why. Stanford may have improved while Harvard declined. Or, Stanford may have improved while Harvard’s quality stayed the same. Or perhaps both schools saw a decline in quality but Harvard’s decline was more severe. In fact, if every single school saw a marked decline in quality the U.S. News rankings would not indicate that this happened. Instead, students can know only relative performance.
Finally, U.S. News inexplicably places every ABA accredited law school on the list of "The Best." The best at what? U.S. News doesn’t say. But it implies that every school on the list is good. The truth is that once costs and employment outcomes are considered in comparison to personal career goals, many schools are bad choices. The U.S. News rankings provide no help in drawing the line.
Rankings are not inherently bad. In fact, they are conceptually quite useful. They order comparable things to help people sort through more information than they know how to or can weigh. However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.
Here is the body:
Tomorrow, the law school world will overreact to slightly-shuffled U.S. News rankings. Proud alumni and worried students will voice concerns. Provosts will threaten jobs. Prospective students will confuse the annual shuffle with genuine reputational change.
Law school administrators will react predictably. They’ll articulate methodological flaws and lament negative externalities, but will nevertheless commit to the rankings game through their statements and actions. Assuring stakeholders bearing pitchforks has become part of the job description.
If the rankings measured something useful, the entire charade would be much easier to stomach. The unfortunate irony is that these rankings adversely affect the decision-making process for law school administrators and prospective law students alike. The stakes are high. Our profession and society need law schools that don’t figure inefficient metrics into annual budgets. Dollars spent chasing U.S. News rankings diverts funds away from students’ education. It also stands in the way of reducing tuition.
In this post, I examine five U.S. News rankings failures. I consider the methodology and underlying rankings theory from the perspective of a student who features job prospects prominently in his application and enrollment decisions. Considering the near universal support for prioritizing job outcomes in the process, these failures demonstrate just some of the reasons annual consternation hardly seems worth it.
Word cloud of LSAT test-takers surveyed about the reasons they plan to obtain a J.D. degree. Source: Above the Law.
First, the rankings pay insufficient attention to what matters most to prospective students: job outcomes. In a survey of 600 students studying for the October 2012 LSAT, Breaking Media’s research arm found that the two most popular words associated with the students’ purpose of getting a J.D. were “career” and “work” (image above). These are not exactly shocking results. Despite the importance of job outcomes, they account for only 18% of the rank and credit schools for jobs few attend law school to pursue.
Second, the rankings use a national scope, which places schools on the same scale. Only a handful of schools have a truly national reach in job placement. The rest have a regional, in-state, or even just local reach. The relative positioning of California Western and West Virginia in the rankings is virtually meaningless. Graduates from these schools do not compete with one another.
Percentages in above graphic are of the entire class.
It turns out that 158 schools place at least half of their employed class of 2013 graduates in one state. The top state destination for each school accounts for 67% of employed graduates. A much smaller 8.2% of employed graduates go to a school’s second most popular destination, with just 4.5% of employed graduates working in the third most popular destination. Only 20.4% of employed graduates (16.7% of the entire class) end up in a state other than the top three. Comparing schools across the country just doesn’t make sense.
Third, U.S. News rankings follow an ordinal system that fails to show the degree of difference between schools. Are Columbia and NYU virtually tied? Or does the two-rank difference represent a wide gulf in quality? Is the so-called difference between Columbia and NYU the same as the difference between Cornell and Vanderbilt? Students weighing school prices need to know not just which school is better but how much better it is.
Fourth, performance changes over time but year-to-year comparisons are virtually impossible using the U.S. News rankings. U.S. News will tell you that Stanford knocked Harvard out of the #2 spot in 2012-13, but the swap in rankings does not provide any clues to your average reader as to why. Stanford may have improved while Harvard declined. Or, Stanford may have improved while Harvard’s quality stayed the same. Or perhaps both schools saw a decline in quality but Harvard’s decline was more severe. In fact, if every single school saw a marked decline in quality the U.S. News rankings would not indicate that this happened. Instead, students can know only relative performance.
Finally, U.S. News inexplicably places every ABA accredited law school on the list of "The Best." The best at what? U.S. News doesn’t say. But it implies that every school on the list is good. The truth is that once costs and employment outcomes are considered in comparison to personal career goals, many schools are bad choices. The U.S. News rankings provide no help in drawing the line.
Rankings are not inherently bad. In fact, they are conceptually quite useful. They order comparable things to help people sort through more information than they know how to or can weigh. However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.
-
- Posts: 12612
- Joined: Mon Oct 03, 2011 12:16 am
Re: US News Law School Rankings -- 5 Failures
Great post.
-
- Posts: 2151
- Joined: Wed Jan 15, 2014 2:18 am
Re: US News Law School Rankings -- 5 Failures
I think your article was great, but I do wish there was a bit more detail about the actual "methodology" that US News uses. I was vaguely familiar with the methodology, but after looking at http://www.usnews.com/education/best-gr ... s-rankings , I now feel even more disgusted by how nonsensical their approach to ranking schools is, and how much weight 0Ls place on them.
Last edited by runinthefront on Sat Jan 27, 2018 12:33 am, edited 1 time in total.
- romothesavior
- Posts: 14692
- Joined: Fri Jun 26, 2009 4:29 pm
Re: US News Law School Rankings -- 5 Failures
Stickied and copied to the Law School Admissions Forum.
Great article.
Great article.
- Dafaq
- Posts: 354
- Joined: Sat Feb 15, 2014 6:19 pm
Re: US News Law School Rankings -- 5 Failures
Albeit, UT is a perennial #15 on USNWR and also #15 on ATL.jenesaislaw wrote:However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.
Want to continue reading?
Register now to search topics and post comments!
Absolutely FREE!
Already a member? Login
- jenesaislaw
- Posts: 1005
- Joined: Mon May 19, 2008 6:35 pm
Re: US News Law School Rankings -- 5 Failures
and...Dafaq wrote:Albeit, UT is a perennial #15 on USNWR and also #15 on ATL.jenesaislaw wrote:However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.
- lacrossebrother
- Posts: 7150
- Joined: Wed Jul 30, 2014 11:15 pm
Re: US News Law School Rankings -- 5 Failures
I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.
I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (http://top-law-schools.com/forums/viewt ... 6#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.
I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (http://top-law-schools.com/forums/viewt ... 6#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.
lacrossebrother wrote:Dumb article.
1. the goal is to come up with some proxy to determine what the best law schools are. anyone can list last year's employment rankings. you'll find however that those numbers fluctuate. the fact that everyone says you want to be a lawyer --that's the point of the lsat --doesn't mean that the fucking purpose of legal education is to maximize employment outcomes only.
what's fucking even more insane is that you started an entire website called "law school transparency" whereby you challenge the notion that a single percentage of employment is an appropriate means to gauge employment . so you fucking say "only 18% of your ranking is based on this really important number..." even though you think it's a shitty number to begin. dope.
moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."
2. i don't even get this point. it's impossible to try to determine if university of arizona is a better school than uconn because people who go to school in arizona won't want to work in uconn? No shit. Perhaps this is why they don't do a fucking 100% employment ranking?? Instead, they ask judges and employers to rate their perception of the products of these schools. They try to rate the faculty. And ya, they take a look at the fucking incoming product because it's pretty easy to see that if one school, on average, has a bunch of dopes attending and the other doesn't, the fact that they aren't in competitive markets doesn't preclude reaching a conclusion. They try to gauge the respective quality of the school's infrastructure and commitment to research by looking at the size of the library (volumes is an imprecise measure, but whatever). The fact that they attempt to do this for everyone, for their magazine, doesn't make the whole thing a failure.
3. There's a raw score you fucking moron
4. it's a failure that the magazine doesn't do a more in-depth analysis of each 2-3 swap? Mr. "only 18%" is upset that the magazine doesn't explain with data graphics somewhere what the most heavily involved coefficient contributing to each swap.
5. ok ya i agree this is dumb. what failures.
- Lawgenius
- Posts: 11
- Joined: Thu May 14, 2015 2:04 am
Re: US News Law School Rankings -- 5 Failures
[/quote]lacrossebrother wrote:moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."
So basically they did a survey accounting only for 18 percent of the rank and expected law students to come over to these schools due to authority and popularity. Obviously the idea is gold, and misleading, and it is funny. 18 percent is a bad number, do we need to argue with this? 18/100 isn’t even 1/2 of the 1/4 of 100. So tell me why it is a good number and i will still not believe.
-
- Posts: 73
- Joined: Fri Jan 09, 2015 6:05 am
Re: US News Law School Rankings -- 5 Failures
you mean you wrote a fucking criticism of this fucking thing in the other fucking thread way back in fucking March. Fuckkkkk. to be fair i fucking agree with you. brolacrossebrother wrote:I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.
I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (http://top-law-schools.com/forums/viewt ... 6#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.
lacrossebrother wrote:Dumb article.
1. the goal is to come up with some proxy to determine what the best law schools are. anyone can list last year's employment rankings. you'll find however that those numbers fluctuate. the fact that everyone says you want to be a lawyer --that's the point of the lsat --doesn't mean that the fucking purpose of legal education is to maximize employment outcomes only.
what's fucking even more insane is that you started an entire website called "law school transparency" whereby you challenge the notion that a single percentage of employment is an appropriate means to gauge employment . so you fucking say "only 18% of your ranking is based on this really important number..." even though you think it's a shitty number to begin. dope.
moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."
2. i don't even get this point. it's impossible to try to determine if university of arizona is a better school than uconn because people who go to school in arizona won't want to work in uconn? No shit. Perhaps this is why they don't do a fucking 100% employment ranking?? Instead, they ask judges and employers to rate their perception of the products of these schools. They try to rate the faculty. And ya, they take a look at the fucking incoming product because it's pretty easy to see that if one school, on average, has a bunch of dopes attending and the other doesn't, the fact that they aren't in competitive markets doesn't preclude reaching a conclusion. They try to gauge the respective quality of the school's infrastructure and commitment to research by looking at the size of the library (volumes is an imprecise measure, but whatever). The fact that they attempt to do this for everyone, for their magazine, doesn't make the whole thing a failure.
3. There's a raw score you fucking moron
4. it's a failure that the magazine doesn't do a more in-depth analysis of each 2-3 swap? Mr. "only 18%" is upset that the magazine doesn't explain with data graphics somewhere what the most heavily involved coefficient contributing to each swap.
5. ok ya i agree this is dumb. what failures.
-
- Posts: 40
- Joined: Mon Jun 01, 2015 1:02 am
Re: US News Law School Rankings -- 5 Failures
The point that schools are spending money to improve rankings when they could be putting it towards lowering the cost of tuition is a good one. I've read many of these articles, but few note that.
-
- Posts: 542
- Joined: Sun Nov 24, 2013 5:13 am
Re: US News Law School Rankings -- 5 Failures
Yeah, it's really interesting to see how much weight is put on factors that, in many cases, won't even matter to students. LST has a system where the applicant picks what is most important to them, and then it rates law schools based on those factors. Personally I prefer LST's system, but the different systems (USN, ATL, LST, etc.) for the most part don't end up showing much difference. T14s remain T14s.runinthefront wrote:I think your article was great, but I do wish there was a bit more detail about the actual "methodology" that US News uses. I was vaguely familiar with the methodology, but after looking at http://www.usnews.com/education/best-gr ... s-rankings , I now feel even more disgusted by how nonsensical their approach to ranking schools is, and how much weight 0Ls place on them.
http://www.lstscorereports.com/state/MA/
-
- Posts: 542
- Joined: Sun Nov 24, 2013 5:13 am
Re: US News Law School Rankings -- 5 Failures
But it feels so much cooler to get a conditional 30k scholarship at a 50k/yr school than to get no scholarship at a 20k/yr school.artistar wrote:The point that schools are spending money to improve rankings when they could be putting it towards lowering the cost of tuition is a good one. I've read many of these articles, but few note that.
Dumb, but isn't that the truth?
- HiLine
- Posts: 210
- Joined: Thu Dec 31, 2009 11:57 am
Re: US News Law School Rankings -- 5 Failures
Yet another year later, still no rebuttal. I feel bad for you.lacrossebrother wrote:I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.
I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (http://top-law-schools.com/forums/viewt ... 6#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.
Register now!
Resources to assist law school applicants, students & graduates.
It's still FREE!
Already a member? Login
- lacrossebrother
- Posts: 7150
- Joined: Wed Jul 30, 2014 11:15 pm
Re: US News Law School Rankings -- 5 Failures
Haha. Don't feel bad. LST has and will always remain a shittily run organization despite its noble intentions. They thrive on the law school is a "scam" concept, but can't see that the best way to fix "failures" is to work with the organizations, not against them.HiLine wrote:Yet another year later, still no rebuttal. I feel bad for you.lacrossebrother wrote:I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.
I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (http://top-law-schools.com/forums/viewt ... 6#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.
Indeed, that a year has passed and US News remains the most important school ranker should make you pity LST. Not me. to be. Long live the king.
-
- Posts: 31195
- Joined: Sat Feb 01, 2014 12:23 pm
Re: US News Law School Rankings -- 5 Failures
Shut up Lax
-
- Posts: 52
- Joined: Sun Apr 10, 2016 4:29 pm
Re: US News Law School Rankings -- 5 Failures
I wonder why tls quit updating the rankings. Anyone know?
-
- Posts: 1
- Joined: Sat Mar 25, 2017 7:58 am
Re: US News Law School Rankings -- 5 Failures
Goldchain wrote:I wonder why tls quit updating the rankings. Anyone know?
I wonder that as well!
Get unlimited access to all forums and topics
Register now!
I'm pretty sure I told you it's FREE...
Already a member? Login
-
- Posts: 2
- Joined: Sat Apr 29, 2017 12:41 pm
Re: US News Law School Rankings -- 5 Failures
Job outcomes for lawyers are dismal. Anyone planning to go to law school these days is frankly, not that bright to begin with. If you want real, solid job opportunities, get a STEM degree. And grow up.
-
- Posts: 31195
- Joined: Sat Feb 01, 2014 12:23 pm
Re: US News Law School Rankings -- 5 Failures
Job outcomes are actually pretty good if you go to a good school.seanmitchell wrote:Job outcomes for lawyers are dismal. Anyone planning to go to law school these days is frankly, not that bright to begin with. If you want real, solid job opportunities, get a STEM degree. And grow up.
- cavalier1138
- Posts: 8007
- Joined: Fri Mar 25, 2016 8:01 pm
Re: US News Law School Rankings -- 5 Failures
Yeah! Anyone not getting a STEM degree and programming stuff into other stuff is an idiot! Especially judges, teachers, lawyers, civil servants, and pretty much everyone else who isn't in a STEM field. Idiots, every last one of them.seanmitchell wrote:Job outcomes for lawyers are dismal. Anyone planning to go to law school these days is frankly, not that bright to begin with. If you want real, solid job opportunities, get a STEM degree. And grow up.
- Deserving Porcupine
- Posts: 66
- Joined: Sun Apr 23, 2017 10:46 am
Re: US News Law School Rankings -- 5 Failures
Yeah, also definitely no oversupply of STEM graduates with grad degrees. You basically get a tenure track academic job just for showing up and smiling nicely. Academia is not fiercely competitive, plus there are all those PhDs whose dream was to be working as a lab tech all along... Dreams really do come true. For. Every. STEM. Major.cavalier1138 wrote:Yeah! Anyone not getting a STEM degree and programming stuff into other stuff is an idiot! Especially judges, teachers, lawyers, civil servants, and pretty much everyone else who isn't in a STEM field. Idiots, every last one of them.seanmitchell wrote:Job outcomes for lawyers are dismal. Anyone planning to go to law school these days is frankly, not that bright to begin with. If you want real, solid job opportunities, get a STEM degree. And grow up.
Communicate now with those who not only know what a legal education is, but can offer you worthy advice and commentary as you complete the three most educational, yet challenging years of your law related post graduate life.
Register now, it's still FREE!
Already a member? Login
-
- Posts: 22
- Joined: Wed Sep 27, 2017 1:24 am
Re: US News Law School Rankings -- 5 Failures
If US News is flawed, what's a better list? How am I to decide how good a law school is, which is better than what and so on?
-
- Posts: 22
- Joined: Wed Sep 27, 2017 1:24 am
Re: US News Law School Rankings -- 5 Failures
This is simply false. I have a degree in "STEM" and have been underemployed since graduating in the Great Recession. I am now switching to law as plan B.Deserving Porcupine wrote:Dreams really do come true. For. Every. STEM. Major.
- UVAIce
- Posts: 451
- Joined: Tue Jul 10, 2012 3:10 pm
Re: US News Law School Rankings -- 5 Failures
Come on man. Unless this is some meta trolling of engineers, you cannot have missed the sarcasm there.Jay2021 wrote:This is simply false. I have a degree in "STEM" and have been underemployed since graduating in the Great Recession. I am now switching to law as plan B.Deserving Porcupine wrote:Dreams really do come true. For. Every. STEM. Major.
-
- Posts: 22
- Joined: Wed Sep 27, 2017 1:24 am
Re: US News Law School Rankings -- 5 Failures
Sorry man, it was really late and it went right over my head. It's a sore point for me because you see everywhere that STEM is a golden ticket and it's not.UVAIce wrote:Come on man. Unless this is some meta trolling of engineers, you cannot have missed the sarcasm there.Jay2021 wrote:This is simply false. I have a degree in "STEM" and have been underemployed since graduating in the Great Recession. I am now switching to law as plan B.Deserving Porcupine wrote:Dreams really do come true. For. Every. STEM. Major.
Seriously? What are you waiting for?
Now there's a charge.
Just kidding ... it's still FREE!
Already a member? Login