Concerns about the Validity of the Ongoing Brony Study

[Music Chart Fan 818]

I saw this post recently on Equestria Daily about this ongoing brony study, for which I've occasionally seen requests for survey takers over the last couple of years or so. If you look at the post itself and the comments (as well as other posts about this study over the years), many people write about this study being "for science" or "scientific" or whatnot. However, I can't help wondering whether this study should actually be considered "scientific", and to what extent the conclusions of the study are valid.

 

I would think that there are certain standard practices that have to be followed with human subject surveys in order for the results of those surveys to be accurate, and in order for the subset of the population which is surveyed to be an accurate representation of the general population being characterized. It seems to me like this study might not follow all of those practices, and if not, then claims which the researchers or other people citing the study make about bronies in general (which assume that the survey takers answered truthfully and represent the general brony or non-brony population) would not necessarily be valid.

 

One big issue is that the survey is voluntary and opt-in, and, as far as I can tell, survey takers seem to be recruited just from making posts asking people to take the survey on a handful of MLP-related websites, as well as perhaps people hearing about it at, for example, brony conventions or meet-ups. By recruiting survey takers in this fashion, the study would leave out anyone who does not frequently visit those MLP-related websites (such that they don't see the posts asking for survey takers) or go to brony cons or meet-ups. As an example, two of my younger sisters watch the show, but essentially never visit "brony" websites, and would never even know that this study exists, even though at least one of them could fit the definition of being a "brony".

 

Even then, some people who do see one of the calls for survey takers won't have the time or motivation to take the survey. When I went to the website where the study is hosted, I found that the current survey there is tens of questions long (including questions which ask for written-out answers) and estimated to take 40-45 minutes to complete, and as far as I know, survey takers are not compensated in any way for completing the survey. Consequently, some people (like me) could see how long the survey is and decide that it's not worth bothering to fill it out. Also, one of the previous studies tried to compare bronies and non-bronies, but how exactly were non-bronies recruited to take a 45-minute long survey for no compensation for the purposes of a brony study? Would those non-bronies who were recruited and willing to fill out the survey representative of the non-brony population to be compared against? So, again, the study would leave out people who don't have the time or motivation to take a 45-minute long survey for zero compensation.

 

Furthermore, with a survey as long as this one is, with no compensation for filling it out, a significant number of survey takers may get "survey fatigue" by the end of it, such that later questions in the survey may not be answered fully or accurately. Also, the survey (or previous surveys) may ask questions about age, gender, race, dating life, social attitudes, etc. which potential survey takers may not feel comfortable answering, and as a consequence, they may give false answers or just stop taking the survey altogether. And with online surveys like this which are essentially open to any anonymous person filling out, are there any methods in place of filtering out, for example, trolls giving intentionally false answers on the survey, people who would fill out the survey multiple times, people taking the survey who aren't in the target population to be studied, etc.? So there are several potential accuracy issues even with the completed surveys.

 

In the FAQ section of the brony study website, the response to a question about what the data has proven most certainly is that "the statistics we use are really complex and there are many factors analyzed". What does this mean? Maybe the researchers feed the survey results into some kind of statistical analysis software, which might then give a bunch of analysis about whether correlations between certain variables are statistically significant. Howver, this could be a case of "garbage in, garbage out", in the sense that if the survey results fed into the statistical analysis software are not representative of the general brony (or non-brony) community, then the output analysis from the software could still not be representative of the brony (or non-brony) community, regardless of the statistically significant correlations that were found.

 

In conclusion, this brony study, because of the methods by which survey takers are recruited and the length of the survey, may be surveying mostly hardcore, dedicated bronies - people who would see one of the calls for survey takers and are willing to put in the time and effort to fill out the survey with no compensation. If this is the case, and if concerns like those I've written about above are not adequately accounted for, then conclusions from analyzing the results of this study may not be representative of the general brony population, and it may not necessarily be appropriate to call this study "scientific". To be clear, I don't have a problem with unscientific studies per se, and I'm not saying that this study shouldn't be conducted or that the results of the study are worthless. I just think that this study and the results of it should not be misrepresented or overstated.

 

What do you think? Are these concerns about this brony study legitimate? Can you think of other potential issues with this study?

 

Thanks for reading!

[Jeric 46,835]

What do I think? I can point to to a phenomenal university to obtain a PsyD if you would so like.

[GuyNamedEarl 1,332]

The thing about surveys is people can lie and skew data. The brony community is very diverse, and not the stereotypical type you see on the Internet.

[TigerGeekGuy 4,659]

I see what you mean, but I don't think you should be overly concerned about this. Pretty much any survey that would attempt to describe a community as diverse, conflicted, and dynamic as the Brony community is bound to run into multiple obstacles and misrepresentations, especially given the route they appear to have implemented. They may refine their data gathering procedures later on or they may not, but I don't think it really matters. If it doesn't really affect you as an individual, I really don't think you need to worry about this at all.

[Sunset188 193]

I see what you're saying, but I definitely think you shouldn't worry about it. I personally wouldn't take the survey because I don't feel like us fans of a cartoon need to be studied at all, scientifically or otherwise

[cider float 2,535]

Trolls doing the survey is still considered data and reflective of society as whole. I doubt this survey is properly run through the proper regulations to make it legitimate sociology community approved. But you can never pinpoint how people are in society because people always change their minds. I don't view sociology like science because it's about human behavior and that's ambiguous.

 

But getting a rough estimate is better than a shot in the dark.

Trolls doing the survey is still considered data and reflective of society as whole. I doubt this survey is properly run through the proper regulations to make it legitimate sociology community approved. But you can never pinpoint how people are in society because people always change their minds. I don't view sociology like science because it's about human behavior and that's ambiguous.

 

But getting a rough estimate is better than a shot in the dark. Fandoms aren't taken seriously as they hardly involve themselves politically.

[Guest]

A study like this that is clearly aimed to benefit bronies so the data is obviously going to be skewed. The people conducting the survey look like they are a bit too involved with the fandom, so naturally they are going to skew the data to favor it, I would say. They are even presenting the data at brony conventions instead of say... News sources or scientific publications. I also noticed that the majority of their research is based upon surveys. No psychological evaluations of any kind, despite all three of the main researchers having Ph. D.s and all three of them having background in psychology?

 

Not to mention their website is probably one of the most unprofessional set ups I've ever seen.

 

Also due to the nature of the brony community, it is very likely that many bronies will attempt to "fool" the survey to make the fandom look better. That's why surveys alone are not a viable research option.

Edited August 19, 2015 by Guest

[IJoinedForPonies 124]

I wouldn't take the survey data etc seriously, they can be easily exploited and griefed.

[Sectus 246]

I wouldn't be too worried about it. I try not to get myself worked up over things such as the ongoing nature of the fandom; for me it has always been the show. I enjoy the show, and I watch it on a regular basis, so my primary concern is as to whether it will continue or not.

[Guest]

Not to mention I am questioning the level of competence of these doctors. Just because they claim they have degrees and have practiced for years, I am noticing a bit that is making me doubt they are properly qualified to conduct a study like this with any actual validity:

 

• They have admitted in their own words that they could not get enough non-brony participants for parts of their research and as a result had 8x the bronies to non-bronies. Despite by their own admission this being problematic, they still went ahead and tried to analyze the data.
• Much of their finds are fairly obvious stuff that was already pretty publicly known such as 86% of bronies being male, a majority of them being single, etc.
• Their website is atrocious for a group that is expected to be taken seriously scientifically.
• The people who have conducted the research by their own admission are bronies, so self interest is going to play a role in biased research.
• Very poor grammar for people who had to take masters level English courses and apparently some of which were teachers.
• I question the level of "research" they did beforehand considering they stated that MLP has a "theme of love and tolerance" when that actually is never stated by the show and simply a motto that was coined in the early 4chan days.
• Only using surveys despite all of them having psychology backgrounds as I pointed out before.
• None of them have a medical doctorate. So none of these people are M.D.s which honestly... I think with a study like this you need at least a few M.D.s
• Their backgrounds are rather wishy-washy. Dr. Edwards never links his credentials, nor do any of the other doctors. It's pretty common practice in the scientific community to assert as to why the people conducting the research are "experts".

I think at the end of the day all this teaches us is that just because someone has a Ph.D. doesn't mean they did amazing at school. Many people with Ph.D.s have been wrong or flat out lied to get the results they wanted for various reasons. That and just because you have a Ph.D. doesn't mean you didn't get a C in all your classes.

[1111 1,284]

I don't see them really going out to report the bad stuff but I do see people trying to use this to reinforce stereotypes later on.

[FlootahBabby 265]

Trolls doing the survey is still considered data and reflective of society as whole. I doubt this survey is properly run through the proper regulations to make it legitimate sociology community approved. But you can never pinpoint how people are in society because people always change their minds. I don't view sociology like science because it's about human behavior and that's ambiguous.

 

But getting a rough estimate is better than a shot in the dark.

Trolls doing the survey is still considered data and reflective of society as whole. I doubt this survey is properly run through the proper regulations to make it legitimate sociology community approved. But you can never pinpoint how people are in society because people always change their minds. I don't view sociology like science because it's about human behavior and that's ambiguous.

 

But getting a rough estimate is better than a shot in the dark. Fandoms aren't taken seriously as they hardly involve themselves politically.

Why not have the first question be a question on the show that only bronies know? It's a simple solution.

[Music Chart Fan 818]

What do I think? I can point to to a phenomenal university to obtain a PsyD if you would so like.

 

Heh heh, thanks, but I'm already working toward a Ph.D. in polymer engineering.

 

 

@@Key Sharkz, very good points all around. I definitely noticed that the website's layout, text formatting, spelling and grammar issues, etc. don't inspire confidence in the scientific rigor of the study. Also, as you might have noticed, the website mentions in a couple of places that the researchers are working on submitting articles to professional journals, but I feel like I'll only believe this is a publishable study when it actually gets published.

[Guest]

Heh heh, thanks, but I'm already working toward a Ph.D. in polymer engineering.

 

 

@@Key Sharkz, very good points all around. I definitely noticed that the website's layout, text formatting, spelling and grammar issues, etc. don't inspire confidence in the scientific rigor of the study. Also, as you might have noticed, the website mentions in a couple of places that the researchers are working on submitting articles to professional journals, but I feel like I'll only believe this is a publishable study when it actually gets published.

Yeah it's just another example of how there are plenty of "scientists" who never amount to anything.

[cider float 2,535]

A study like this that is clearly aimed to benefit bronies so the data is obviously going to be skewed. The people conducting the survey look like they are a bit too involved with the fandom, so naturally they are going to skew the data to favor it, I would say. They are even presenting the data at brony conventions instead of say... News sources or scientific publications. I also noticed that the majority of their research is based upon surveys. No psychological evaluations of any kind, despite all three of the main researchers having Ph. D.s and all three of them having background in psychology?

 

Not to mention their website is probably one of the most unprofessional set ups I've ever seen.

 

Also due to the nature of the brony community, it is very likely that many bronies will attempt to "fool" the survey to make the fandom look better. That's why surveys alone are not a viable research option.

And then use an unpublished source to make their point when arguing with anti-brony bots

[Nursing Major 1]

Well, Key Sharkz, I can validate Dr. Edwards' education. He is my college psychology professor and actually got our entire class to participate in a part of the study. This was studying people's initial reaction to hearing about bronies. Once he explained it to us he gave us a sheet of paper that had questions like "what age do you think these men are" and "what do you think their sexually orientation is". He gave a personal lecture to the whole class and was very official about the whole ordeal.

[Music Chart Fan 818]

Well, Key Sharkz, I can validate Dr. Edwards' education. He is my college psychology professor and actually got our entire class to participate in a part of the study. This was studying people's initial reaction to hearing about bronies. Once he explained it to us he gave us a sheet of paper that had questions like "what age do you think these men are" and "what do you think their sexually orientation is". He gave a personal lecture to the whole class and was very official about the whole ordeal.

 

It's interesting to hear from an actual student of Dr. Edwards, so thanks for sharing. If you don't mind answering, when you say he was "very official about the whole ordeal", do you mean that he explicitly informed the class that your responses would be included in this ongoing brony study? Did he allow opportunity to opt out, inform you about non-disclosure of your personal information in the study, etc.? Did he, e.g., check whether any students considered themselves bronies or check whether students had heard of the brony phenomenon (and already formed an opinion about it) prior to that class? I'm curious whether responses from his classes might actually become part of the "official" brony study or whether your in-class activities are just informal exercises in a topic in which he's interested.

[Nursing Major 1]

He did say that we could opt out and that the results would be added to to the study. He did ask if anyone had heard of bronies before now and most of the class raised their hands. He said that when he first started the surveying that only a few people said that they had heard of them. He also said it was best if people had NOT heard of bronies before, because the point of the survey we took was to get our first reactions to the brony fandom.

[Commissar Alexer 763]

A study made by bronies about bronies and mostly for bronies.

 

Sounds legit.