James Maguire at Datamation (the oldest IT publication this side of a PDP-11 manual) pointed me to glassdoor.com and asked what I think of this new web site that gathers and reports salary information in the information technology industry. His article is a fun read: IT Salaries: Glassdoor Reveals Tech Pay Figures.

This web site — funded by Benchmark Capital and run by guys from places like Expedia who oughta know better — collects tech-industry salary information from anyone who submits it. Maguire didn’t tell me this, but I heard a rumor that every morning a dog appears at glassdoor.com with a note in its mouth, and is promptly fed, stroked, and sent back out to fetch more data…

Come on, guys. This is ridiculous. But I’ll bite. Having information about who’s earning what at which companies could be useful, as long as you don’t let it effectively cap your own worth. Of course, glassdoor.com needs to make it over one big hurdle: Is the salary information it publishes legit?

Maguire asks this one question every which way, and my hat is off to the guy for keeping a straight face. And here’s a sample of the answers he got:

“We’ve got a bunch of technical and procedural mechanisms to vet the data.” When pressed, one of the founders demurs, “We’re not really talking about the specifics that we’re using.” (Cough-cough.)

Even Ambrose Bierce, owner of a recipe once more closely guarded than the formula for Coca Cola, revealed more than that, in Oil of Dog.

Let’s continue:

How does glassdoor.com know the salary figures that folks send in aren’t total doody? Chief exec Robert Hohman says, “We apply a ton of statistics to it.”

I figure every company must guard its secrets. But glassdoor deserves a bone for for statistics doubletalk: “We apply standard statistical analysis to make sure the data looks about right.”

About right. Sheesh. (These guys started Expedia?) Gimme more. Maguire must have been scratching this dog awfully low to elicit details about the secret, scientific methodology: “The front line of defense is a human – nothing replaces a human sitting down, looking at the data.”

Yah, I learned that at Rutgers, in my first stats course. Looking at the data while sitting down improves the quality of the data. I wish I could have watched Maguire’s expression (did he arch his eyebrows?) when he probed further about the “filtering” of data that glassdoor does. “Ultimately, we review every single review by hand,” Hohman said. “So everyone passes through a human being and we check for various criteria to make sure it meets community guidelines.”

Glassdoor’s own hires must be rigorously tested for intestinal fortitude. But, what do  community guidelines have to do with data quality?

With all the thousands and thousands of salary figures and company reviews streaming in, Maguire asks the glassdoor crew whether they’ve got sufficient staff. “There’s probably about a dozen of us,” Hohman explained, noting, “We’re still figuring that out.”

Must be a head-count problem, but nothing some additional funding from Benchmark won’t fix. The key to any venture group is that glassdoor has “made a commitment to data integrity.” Carry on, Men! (Women, too, but no women are mentioned. I think they could use some women.)

In the end, how do they know the salary figures people send in are accurate and truthful? “We compare companies’ data one to another to make sure the data looks right.”

Looks right. Scritch-scratch. Scritch-scratch. Maybe they should have that dog stick around to sniff out the anomalies… I was wondering how Maguire did this interview with a straight face. Then I noticed his smirking photo at the top of the article. But I stopped shaking my head after the statement, “…There’s probably about a dozen of us…”

Give James Maguire a hand for keeping his sense of humor. Don’t bother taking this salary dog for a walk. (These guys at glassdoor.com need to get an abacus before Benchmark Capital sits down and looks at the data by hand…)

8 Comments
  1. Thanks for covering this, Nick. I recently read an article on Computerworld about Glassdoor and found myself wondering about the quality of the data.
    http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9096718

    Here is a digest of tech blogs about Glassdoor…
    http://blogs.computerworld.com/salaries_through_the_looking_glassdoor_com

    I’d really like to read your take on the article linked below. Perhaps in a future post?

    Why women quit technology careers
    http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=319212

  2. There are certainly some good data there. One could generously even imagine that they constitute the majority. Just like every good lie contains a little truth.

    I don’t doubt that glassdoor has good intentions, because most companies prefer to sell a product of intrinsic value if given the choice. But you can’t clean up self-reported, sensitive, confidential data and make them meaningful…and the very act of cleaning them up makes the enterprise even more of a shot in the dark.

    Job hunters are better off guessing the applicable salary ranges themselves.

  3. Hi,

    I got to read this and wanted to share with you.

    Even CEOs think CEOs are overpaid?

    BOSTON (Reuters) – With many U.S. chief executives taking home millions of dollars in pay, it is no shock that average workers regards them as overpaid. But that attitude extends to the corner office as well.

    Sixty-four percent of top executives view CEO compensation as excessive, according to survey released on Tuesday.

    The poll of 1,572 readers of BNET.com, a business Web site, found that, overall, 77 percent of employees regarded CEOs as overpaid. The survey, of readers of the Web site, was conducted June 11 through 18. About 90 percent of respondents were from the United States.

    Fifty percent of CEOs surveyed said their leadership style was effective, but only 38 percent of employees agreed. BNET.com is a unit of U.S. media technology company CNET Networks Inc.

    (Reporting by Scott Malone, editing by Gerald E. McCormick)

    Regards
    Amit

  4. Mike B.: Bingo! You can’t “clean up” data which by definition is questionable. If “humans” are “confirming” it, as glassdoor claims, then the source of confirmation is sufficient. Why bother with questionable submissions from readers to begin with?

    I agree that we should presume a company wants to provide a product that has intrinsic value. But I think glassdoor is trying too hard to put a spin on a simple idea. Credible trade publications (E.g., Computerworld) regularly conduct salary surveys. This makes glassdoor superfluous – except that it also provides a forum for dissatisified people to trash their employers. And that’s what suggests our presumption is not justified in this case.

    People can get better data from better sources than glassdoor. Please note that I don’t believe ANY salary data is truly useful; it’s more distracting.

  5. Hi Nick – funny post, you did catch me on about my 500th interview and I didn’t use perhaps the best choice of words in describing our review process. That doesn’t, however, diminish the seriousness and commitment we have around our data review process.

    What is different about glassdoor.com is that you can see salary data at the *company* level, unlike ComputerWorld – or any other salary source that we could find out there – which aggregates to regional levels at best.

    I think you summed it up best yourself – you don’t believe that ANY salary data is useful. I happen to disagree. I fundamentally believe that people have very poor information right now, and giving them access to this salary data helps them in negotiating.

    I’ve personally spent a lot of time talking with company CEOs, HR reps, and both in-house and external recruiters as part of Glassdoor’s development process and would love to set aside some time to connect with you to share a bit more about our process here and future plans. I think your feedback would be extremely valuable. You have my email address, we should talk.

  6. Robert,

    How does anecdotal salary information whose source and quality are unconfirmed constitute valid “salary data”?

    It doesn’t matter how you review it, process it, vet it, or handle it if its validity is entirely questionable from the start. You can’t run statistical analyses on questionable data and claim useful conclusions.

    If you were gathering salary “at the company level”, that would be an entirely different story. But you’re not.

    I agree with Mike B. that your intentions are probably good, but I also agree with him that “cleaning up” such data is suspect at best.

    A focus on salary data by an employer or a candidate results in negotiations that drive a job offer toward the mean salary, not the right salary.

  7. I think we’re going to have to agree to disagree Nick. I don’t think giving employees this information drives salaries towards the mean, I think it arms them with exactly the data they need to level the playing field in a tough negotiation.

    I welcome the opportunity to talk with you directly about the work that we are doing, and forthcoming features which will make it even easier for people to get this important data.

  8. Robert, I think the bigger question remains. What is the value of salary “data” whose source may be unknown and whose veracity is questionable?