Tired of being left out

I am so tired of Native Americans being left out of diversity studies that I did something about it. I questioned it.

Here is what prompted me. I saw a story today in the Wall Street Journal, which said that Twitter has more diverse users than Internet users. I was hoping to find information about Twitter usage of Native Americans, but was disappointed to learn that “diversity” meant “Black, non-Hispanic” and “Hispanic.” The only other categories were “White, non-Hispanic” and “Other, don’t know, refused.”

Let me be clear. I am a huge fan of demographic studies done by such respected organizations as the Pew Research Center. The one cited in the WSJ story was done by the Pew Research Center’s Internet and American Life Project. I wrote to leaders of the project via email and was pleased to get responses, even though they were not the responses I wanted.

In my initial inquiry, I asked whether I could get data on Twitter usage of Native Americans. In addition, I asked for an explanation of why the sample size of Native Americans are considered “too small.” I received this response from Aaron W. Smith, senior researcher at Pew’s Internet and American Life Project:

“[W]hile it may sound like a cliché, you are in fact correct that we do not report on Native Americans as a distinct group due to sample size limitations. To use the example you mentioned, the survey that the WSJ article was based on included a total of 1,801 respondents–but only 19 of those 1801 respondents identified themselves as Native American. That means that any findings we reported for that group would have a margin of error of plus or minus 26 percentage points attached to them, which is obviously far too large a range for us to feel confident reporting those findings. Believe me, we would love to report stand-alone findings for Native Americans (and for the many other relatively small groups that we are currently unable to report on–Jews, Mormons, LGBT individuals, Asian Americans, the list goes on and on). But the unfortunate reality is that reaching those groups in sufficient numbers is very expensive and we have a very finite research budget to work with.”

I was also given a link to an explanation about why the Pew Internet project does not include Asian Americans in its study on technology, which also applies to Native Americans and other small sample-size groups. Click here to read it.

In a subsequent email exchange, I asked what the cost might be to expand the “reach” to include Native Americans in further studies. Here is Aaron Smith’s response:

“And as to cost, I don’t have an exact dollar figure but my understanding is that it’s quite prohibitive. Some of my colleagues examined this question in the past and quickly realized that doing one survey on basic ownership, usage, and adoption metrics with the required number of respondents to report on Native Americans, Asian Americans, and some of the other small-incidence groups I mentioned earlier would effectively exhaust our entire research budget for the year–meaning we wouldn’t be able to do anything on health, privacy attitudes, teens, politics, the role of libraries, or any of the other subjects we cover. It’s possible that we will be able to increase some of that small-group coverage as we potentially move to greater usage of online surveys in the future, but at least for the moment phone polling is our bread and butter for a variety of reasons.”

I appreciate very much Aaron Smith’s quick responses. It’s helpful to know the limitations. Now I want to find out how standard research methods can be improved upon so that it is easier to gather data from groups that are considered small sample-size groups under the current methodology. Any ideas?

Follow me on Twitter @karenmichel.


12 thoughts on “Tired of being left out

  1. Good post, and as you know, I think this issue is a tough nut to crack. Seems that statisticians might know some analytical methods for drawing at least some conclusions from small sample sizes. Maybe there are different techniques that could be used in the design of future surveys–like “clustering” perhaps. I just read about clustering in a story about an Esquire-NBC News survey that shows that there is a “New American Center” in politics and that we aren’t as divided as we’ve been told.

    Here’s what the story had in there about the survey methodology:

    The Benenson Strategy Group and Public Opinion Strategies conducted a nationwide survey from August 5 through 11, 2013, with 2,410 registered voters. They applied a k-means clustering technique to group respondents into “segments” based on attitudinal and demographic commonalities and like-mindedness. They conducted eight iterations of the clustering to optimize the differentiating variables that feed into the segmentation methodology. The segments were formed based on commonalities across their demographics; psychographics; political, social, and economic values; and lifestyles. The pollsters selected the segmentation solution that yielded the most unique and differentiated clusters.

    OK, I have no idea what “k-means clustering” means … but apparently there are techniques out there that perhaps could be used on future surveys to gain insight on the opinions of people from relatively smaller or otherwise tough-to-sample segments.

    Any statisticians out there in the know when it comes to overcoming the “small sample size” issue when examining the views or opinions of smaller ethnic groups?

    Roberto Michel

  2. I can comprehend how Native Americans are continually disqualified by technology. I would just like to see that Natives are counted and what “small” response was retrieved in surveys. It may be seem automatically non-material to the greater sound, but I would like to witness at least being thought of in surveys/studies/statistical information. Let our small spot on the population picture be seen – even on a wee scale. Hopefully, we can use the information.

  3. Roberto: Thanks for your input. I’ve heard of clustering, but I don’t know much about it. It’s a good avenue to check. I can find out more and, hopefully, make a follow-up blog post.

  4. Hi, Ronnie. Thanks for your comment. By the way, Aaron Smith gave me one statistic on Native Americans from the Twitter survey, but he told me take it with “the necessary grains/boulders of salt” because of the small sampling. He said 12 percent of Native American Internet users in the survey said that they used Twitter, compared with 16 percent of all adult Internet users. Keep in mind that, because only 19 Native Americans responded, the reliability of the data is plus or minus 26 percentage points.
    I was glad that I asked for the data, even if that statistic is considered unreliable.

  5. This is an important issue that causes us difficulties every day. The best example is the U.S. Census. The Census collects data but expanded the American Indian category to cover almost all other races, without asking the question of tribal enrollment. When I speak at colleges and universities, and they ask me what they can do for us, I always say include us in the studies and include us in the data. There are ways to do it and they and we must keep on trying. Thank you Karen for this great blog piece!

  6. Thanks, Dorene. I need to talk to a demographer about the most efficient ways to collect data from Native Americans and Native American communities. There needs to be a push for getting greater diversity in data collection so the “small sample-size groups” won’t continue to be left out.

  7. The problem is basic demographics. If the sample size is 600 people, then a reflection of Native Americans would be 3. That’s not enough to do statistical analysis. Even 6,000 would not be enough. More like 60,000 (and that’s not enough to break out tribal responses for the same reason). The American Community Survey which reflects Indian Country pretty well has an annual sample size of 3 million. There is a fabulous look at polling problems in elections from Geoff Peterson in his academic study, “Native American Turnout in the 1990 and 1992 Elections.

  8. I remember once being told by a school official that the reason why there were no statistics on Native American graduation rates/graduation test scores was that the group was “statistically insignificant”. My response to him was “So the American plan to commit genocide is finally working?” He looked at me funny and I replied, “You can come to my house and tell my Native American children that they are insignificant, if that’s what you think. They know that they are the most important people in the world and they will laugh after you leave (they would never laugh in your face). I am not sure I will be able to convince them that a high school diploma is worth having after you leave, but just in case I do, you should plan on ordering their caps and gowns in the appropriate sizes, because we will certainly attend the ceremony.” He had absolutely nothing more to say, and that was a good thing, because I wasn’t interested in hearing any more. Statistics are valued by Caucasians, individuals by Natives. Every Native person deserves to be asked their opinions and counted in every survey. Surveys are an artificial way of gaining information anyway…just my opinion.

  9. I totally agree with you, Karen. Native people seem to be excluded more than included whenever there are diversity issues. While it may be that some consider us to be “too few to be counted,” we still have a voice. Thanks for your blog and making our voice heard.

  10. Mark, you make some good points. I am researching alternative ways to compensate for small sample sizes. I am hoping to make a follow-up blog post on this subject.
    Leslie, thanks for sharing your story. For too long, I have accepted the reasoning that the Native American population is statistically too small to compare with other groups. Native Americans matter, and we have to find a better survey method to include them in important studies.
    Lori, thanks so much for joining in the discussion. You’re so right. We have a voice, and we need to make it heard.

  11. While qualitative data compiled from focus groups and interviews cannot be generalized, it is still data and that is being collected every day. Researchers in NSF and many health grants use these techniques.

    • Hi Dorene. Somehow I missed this comment you submitted weeks ago. Sorry about that. I agree that qualitative data is very useful. We need to gather data in various forms. I’m more of a hard data person who likes to crunch numbers and see what kinds of stories they tell.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s