Results are In: Where are the Women?

Today A List Apart came out with a survey of the internet industry, working conditions and what not (http://www.alistapart.com/articles/2007surveyresults) . They claim 33,000 respondants, "Close to 33,000 web professionals answered the survey’s 37 questions, providing the first data ever collected on the business of web design and development as practiced in the U.S. and worldwide."

Well, I have news for them: Self-reporting surveys get skewed in funny ways. There's a reason the census goes house to house (besides Congressional screwyness). There's a reason Gallop cold calls people. There's a reason scientifically valid surveys use random sampling.

And, no matter how many people you email it to, any survey that is open on the web is not a statistically valid sample. It's called selection bias, and it totally screws your results. A good example of this would be the nightly call in poll on Lou "I hate immigration" Dobbs. (http://www.cnn.com/CNN/Programs/lou.dobbs.tonight/. You will note the disclaimer, "this is not a scientific survey.") Or, American Idol voting. The people who think everyone on American Idol should be locked away in an institution somewhere in order to raise the general lucidity of society? They don't take the American Idol survey.

In the end, this survey is not the be all and end all they are touting it to be. It's biased. It's biased to people who read Zeldman and his friend's sites. And that's NOT every web designer in America. I, for example, don't read Zeldman. I know, I'm supposed to, but his writing style leaves me cold. The editorial style of A List Apart just doesn't do much for me. This doesn't mean I don't have a place in this field, but it does mean I'm less likely to read anything on A List Apart.

Yes, 33,000 is a lot of people. But it's a self-selected non-scientifically based sample. It's a good starting place.

This field is NOT actually only 16% women. That only 16% of Jeffery Zeldman's avid readers are women? THAT I would believe.


Anyway: Alistapart's survey of the field (who is doing what where) was posted online:
http://www.alistapart.com/d/2007surveyresults/2007surveyresults.pdf

Ok, kudos to A List Apart for trying. But if you're going to try and claim some sort of moral superiority for your results, you should hire a professional survey company.

People on the comments of this article are using it to JUSTIFY gender skewing at conferences and in hiring. Hello, it's a self-reported survey. It's not Gallop! It's not the census! It's going to be scientifically INVALID.


-------------------------
Suggested reading if you for a minute think a self-reported survey on the web justifies anything: (Some of these may be old, but they are still in print for a reason!)