science Word frequency analysis of all episodes so far

[SasQ 1,376]

Hi.
If anypony interested, I made a list of all words appearing in every single episode of MLP aired to this day (September 4th, 2017), grouped and sorted by frequency of appearance (that is, how often it appears in the show). Here's the link to the text file with the list:

http://sasq.comyr.com/Stuff/Jajca/QC/Freqs/All_freqs_merged.txt

I'll try keeping the list up to date with upcoming episodes (if I'll have enough time), or at least after every season.

"What's the use of such a list", you may ask?

Well, beside knowing what English words appear the most often, one can also check how many times a certain pony is mentioned by name, or how many times something else is mentioned, or if it is mentioned at all. One can also see what words are the least frequent and try to find out why (maybe there are some scientific reasons behind it? who knows...).

The file is automatically generated through a bit of Bash scripting voo-doo magic (don't worry, I'm not that crazy to do it by hand! ), so maybe I'll try to make some comparisons later about how the word distribution changed from season to season (I predict that there might be a big change of the frequency for the word "elements" and "harmony" after Season 4 premiere, we'll see if I'm right... ;-J ), or maybe which characters had some more spotlight than the others.

For example, the word "twilight" appears 846 times and it is 65th most frequent.
The word "pony" appears 665 (almost satan, heheh :q ) and it is 88th most frequent.
The word "princess" is just three positions lower (91st), being used 652 times.
The word "friends" appear 650 times, and it's 93rd most frequent. "Friend" appears 376 times. "Friendship": 337 times.
"Rainbow" is 95th and appears 625 times.
Next one is "Pinkie", 624 times.
"Spike" closes the first hundred, being 98th most frequent and appearing 610 times.
"Rarity" is 115th, appearing 502 times.
"Applejack" is 125th, being called by her name 459 times.
The last pony from the main six is Fluttershy: 439 times.
Celestia is 218th, mentioned 204 times. Luna is 299th, mentioned 101 times.
Starlight is 233th, appearing 180 times.
"Ponyville" is mentioned 451 times. "Equestria" 370 times. Canterlot: 164 times. Manehattan only 65 times though :q
The word "cutie" appears 499 times.
The word "everypony" appears 491 times.
The word "apple" is used 473 times, being 122nd on the list.
Interestingly enough, Rainbow Dash's favourite word "awesome" appears the same number of times as Twilight's favourite word "spell": 175 times :q  So why is it that when Rainbow Dash says her favourite word, pretty much everyone picks it up and finds boring/repetitive/overused, while nobody is complaining about the word "spell" when Twilight (or anypony else) uses it? :q  Psychology is weird...

There's one more reason I made that list: I wanted to know which words appear most often in the show, so that I knew what words should I make first in my Unicorn Language

I have a little problem with it, though, so if anypony has some ideas about how to solve it, I'm all ears
The problem is that at the end of the file (well, not just there, but the end of the file is cluttered with them the most) there's a lot of words that aren't exactly words, but some onomatopoeic strings of letters, like "aaaah", "woo-hoooo", "mmhmm" etc., or things like "omigoshomigoshomigosh" or "yesyesyesyesyes" etc. – things that I'd like to get rid of. But it seems that it's not that easy to get rid of them   Sure, the easiest thing would be to remove words that have any letter repeated more than twice. But that would remove only the most obvious crap. Things like "mmhmm" still follow a "correct" pattern of English words – if it accepts things like "commemorate", it will also accept "mmhmm" :-/   No idea about the "yesyesyes..." stuff, though   I can remove them by hand, of course, but that would be veeeery time-consuming and a waste of time anyway, since after a new episode comes out, I would have to repeat that process again after regenerating the files

One idea that comes to my mind is to use some kind of a black list and add all problematic words to it by hand. I would have to do that only once, and then just make another script that would remove every blacklisted word automatically every time the list would have to be refreshed. But maybe there's a better way I haven't thought of?

Edited September 3, 2017 by SasQ
Added some stat info.

[Mirage 24,770]

Wow - 'twilight' is way up on the list, the first noun and first name. Obviously the focus of the show. 'my' and 'little' and 'pony' are also way up there. 'Princess' is also high on the list.

Very interesting!

Once you see all the mane 6 names, then the words start to really imply what kind of show it is specifically (more names and nouns). Words used more frequently than 'twilight' are probably among the most frequently used words in the English language anyway.

[SasQ 1,376]

Yup, pretty much   I added some statistics to my previous post, to let you see some of the patterns more easily.

The top of the list is not very surprising, of course: the most common English words, such as the articles ("a", "an", "the" – they come along with each noun, so no wonder they appear frequently), personal pronouns ("I", "you", "he", "she", "it" – they replace nouns), or prepositions ("in", "on", "at", "from" etc. – they describe relative position), or the "helper verbs" ("is", "are", "be", "can", "have", "has", "had", "will", "go" etc.).

Then it starts getting interesting. As for Twilight it was pretty obvious, since she's the star of the show since its very beginning, and only recently being overshadowed by you-know-who The rest of the mane six also appears pretty often, since they are… well… the main cast :q

Then of course words such as "friend", "friends", "friendship", "magic", "spell", "pony", and the favourite horse puns, like "somepony", "anypony", "everypony" – those will be pretty frequent as well.

There's also quite interesting to see what words appear no more than once. Taking out the onoatopoeias and other clutter, there's a bit of interesting stuff inthere

 

Edited September 3, 2017 by SasQ
Mentioning the edit.

[碇 シンジン 27,431]

I used to make these stats too they are pretty interesting though now I been on hiatus mainly because I have not what it takes to count every characters screen time by watching the episodes and the one whose stats I used hasn't updated his page  I figured that it wouldn't keep me into it to manage just the word counts though I didn't use scripting back then, would probably get the stats without a hassle with it though  writing such a script could be tougher than my current capabilities  allow me to write.

I also believe that gathering the data manually or semi manually is part of the fun  and  finishing it feels lot more satisfying:)

[Mesme Rize 15,683]

Damn, that is really impressive.

It really shows where the focus on the show lies, like Twilight being the true focus of the mane 6 and that the word princess is also quite important.

Great job.

[Jeric 46,851]

I finally looked up instances of 'darling' vs 'fabulous' and darling wins out between the two.

 

[碇 シンジン 27,431]

Also I don't feel as determined to maintain the stats if I'm not able to completely obtain all the needed data as I don't like incomplete stats

[SasQ 1,376]

1 minute ago, ooReiko said:

I have not what it takes to count every characters screen time by watching the episode

I thought about that too, and I think it might be quite possible to estimate the screen time without watching the episodes: One can use the transcripts and extract the character names that precede each of the lines they speak on screen, and then count these. It won't be precisely the time they appear on screen, but it might a be pretty accurate estimation because when a character talks, he usually appears on the screen too. If counting the number of script lines weren't accurate enough, one could also count the number of words spoken in a line, and convert it to a rough estimate of time. All of this can be done using my scirpting voo-doo so maybe I'll try that later too.
Just a thought

2 minutes ago, Mesme Rize said:

and that the word princess is also quite important

As much important as "friendshp", it seems! It appears the same number of times than "friendship", plus two But I need to check how it was before Season 4 finale, because since then we have a "princess of friendhip", and this might cause both words to appear the same number of times.

4 minutes ago, ooReiko said:

Also I don't feel as determined to maintain the stats if I'm not able to completely obtain all the needed data as I don't like incomplete stats

Sometimes a rough estimate can be useful and enlightening as well ;>  (when the sample is over-the-top anyway, for example :q )

[碇 シンジン 27,431]

Just now, SasQ said:

I thought about that too, and I think it might be quite possible to estimate the screen time without watching the episodes: One can use the transcripts and extract the character names that precede each of the lines they speak on screen, and then count these. It won't be precisely the time they appear on screen, but it might a be pretty accurate estimation because when a character talks, he usually appears on the screen too. If counting the number of script lines weren't accurate enough, one could also count the number of words spoken in a line, and convert it to a rough estimate of time. All of this can be done using my scirpting voo-doo so maybe I'll try that later too.
Just a thought

As much important as "friendshp", it seems! It appears the same number of times than "friendship", plus two But I need to check how it was before Season 4 finale, because since then we have a "princess of friendhip", and this might cause both words to appear the same number of times.

Sometimes a rough estimate can be useful and enlightening as well ;>  (when the sample is over-the-top anyway, for example :q )

Well In my stats I already had word counts character counts, sentence counts and line counts for each character in each episode so I thought about converting those into estimated speaking time and use it as screen time but  the characters can speak without appearing on screen so and there are also other factors that make it tough. But as the stats grew and the episode 100 happened background characters came into play too so time on screen is kinda essential as most characters that appear on the screen rarely talk if they talk at all. 

Also I strive for accuracy so I try to be as precise as possible. 

Hmm actually I might update my stats through season 6 now you got me inspired again

[Mirage 24,770]

'Slice' and 'Thud' only used once a piece. Guess only one horror show? 

And no instance of 'hand'.

[SasQ 1,376]

10 minutes ago, ooReiko said:

Hmm actually I might update my stats through season 6 now you got me inspired again

Cool   Can you share the link to the not-so-up-to-date ones too? I'd like to take a look at them some time

Just now, Mirage77 said:

And no instance of 'hand'.

I swear I've heard that word, or similar, mentioned in the show a couple of times, to my big surprise :q  Let me see…

Yup, it's there, 12 times, and 3 appearances of "handy", and 1 of "hands". Also 1 appearance of "hand-stitched" and "handiwork"  WTF? Do ponies import clothes from the Human World? :J  There's also one "handful".

[Mirage 24,770]

omg...I'm a dufus. I didn't start my search high enough on the list. Doh!

[碇 シンジン 27,431]

1 hour ago, SasQ said:

Cool   Can you share the link to the not-so-up-to-date ones too? I'd like to take a look at them some time

I swear I've heard that word, or similar, mentioned in the show a couple of times, to my big surprise :q  Let me see…

Yup, it's there, 12 times, and 3 appearances of "handy", and 1 of "hands". Also 1 appearance of "hand-stitched" and "handiwork"  WTF? Do ponies import clothes from the Human World? :J  There's also one "handful".

Well the stats covering the show to season 5 end are in my deviantart  though I've broaden the stats a lot after that added writers and voice actors etc. 

Something MLP stats

The link to the file should be in the comments somewhere as someone noticed that da download didn't work 

[Ashen Pathfinder 16,161]

@SasQ

As this topic is dealing with FiM it has been moved to Show Discussion. Thank you.

[SasQ 1,376]

Dang it, that's where I wanted to post it in the first place, but then I thought to myself: "Well, it's not exactly that much about the show itself, it's more about statistics and word frequencies and all that geeky stuff, so I better post it here or they'll kick me around again." Well, silly me...

[DonMaguz 1,023]

First of al, awesome job @SasQ it's interesting enough to find the more mentioned ponies and places and with this work is fairly easy to check it out.

 

Now, looking for the word "nightmare" (104 times said), I found something interesting at the words used 99 times...

"trixie took team rock baby" - Now I want to take team rock baby too! 

 

Anyways, I found it funny that these words actually form a phrase, if any of you find out another instance of a phrase or sentence on the file I would like to check them out.

Cheers!

[SasQ 1,376]

@DonMaguz hahah yeah, I noticed that too, and I even played with it for a while But I didn't mention it because as the frequency drops down, more and more words start to have the same frequency, and the more words are there in a line, the more likely they could be assembled into a meaningful sentence. So nothing surprising here, but now when you mentioned it, maybe I'll post the results of my own playing with these words Because when you switch them around, some of them almost look like some kind of subliminal messages or secret foreshadowing heheh

  650 want friends
  517 maybe dash
  491 everypony best                        [true…]
  440 should find
  429 she's better
  368 let's love  
  362 very first                                     [gotta love those YouTube comments… :q ]
  268 whoa together                             [we have to do it together. then we'll open the Harmony Box]
  240 wanna done without                   [and they returned the Elements to the Tree]
  239 ugh pretty                                  [Pinkie: "Heeey! That's my line!"]
  203 wanted hard important
  199 another long exactly
  189 happy old kind
  185 true problem, anypony ?                [asked Twilight in "Lesson Zero"]
  184 heh, apples …                               [another apple bucking season, AJ?]
  180 wasn't starlight today already ? 
  179 remember, hear amazing hope
  178 each special                            [yup, each pony is special on its own way ;> ]
  175 ah getting awesome spell       [well, good for you :J ]
  174 supposed worry
  173 sister care cake bloom       [is that about one of those "I care" pancakes Celestia made for her sister? ]
  172 ya happened, granny        [yup, you certainly did ]
  163 y'all talk, girls                     [yup, you definitely do :q ]
  159 stay through                      [and watch pone]
  154 found guys                        [good for you]
  153 once start discord             [yeah, me too]
  147 sweet myself                     [cool, just don't eat yourself ]
  143 miss understand               [that would be a cool surname ]
  136 mind yourself                    [I always do :q ]
  135 town 'em                           [how? the Starlight Glimer's way?]
  131 win whatever                      [done]
  130 haven't heard talking                  [then clean up your ears maybe? :q ]
  129 many aren't listen heart              [yup, especially in the Human World]
  125 play empire                                 [already played]
  122 hmm needs makes alright
  120 aw, used                                    [I feel bad for ya :q ]
  119 having hooves                          [me too!]
  118 quite chance
  116 nopony glad belle until yet           [yeah, she's a bit underappreciated character :J  but it's getting better after she learnt how to magic ]
  113 wonderbolts stuff gala, sweetie: eeyup, crusaders!       [the Crustaders at the Gala hanging up with Wonderbolts? ]
  111 wish daring matter                    [yeah, I wish that too]
  110 thinking head happen                 [that's always good]
  106 wants powerful saw                    [I'm scared to ask what for? :s ]
  105 favorite mac learn somethin different      [I remember that episode too. Big Mac in a drag cannot be forgotten :J ]
  101 finish plan luna                       [roger that]
  100 elements world though
    99 trixie rock team took baby       [you mean that one from "Rainbow Rocks"? ;> ]
    98 excited finally, wonderful excuse, cadance almost dress      [glad that you enjoyed the episode and the fact that the writers finally found a wonderful excuse for Cadance to almost dress up ]
    97 cider trouble                            [the only trouble with cider is that there's always too few of it :q ]
    96 learned cool games                    [the games ponies play?]
    95 hearth's story comes easy, seems        [yup, especially when you rip off "A Christmas Carol" ]
    94 least power                                              [with least power comes least responsibility ;J ]
    93 suppose maud says silly, afraid map      [Maud always says silly, but no sillier than her pink sister ;> ]
    92 lost moon                                               [I bet Luna stole it!]
    91 surprise job days                                  [yeah, I remember these days too, when I was unemployed]
    90 grand funny face
    87 tried hoof doin, darling                         [did you go to the spa again, Dashie?]
    82 super strong ain't read                        [unless it's Daring Do books ]
    81 sparkle brother goodness                   [said Cadence ]
    79 uh-huh absolutely awful music           [who gave the fluegelhorn to Pinkie again? :q ]
    77 we'd stand hundred                           [against the season 7 finale's boss, I guess?]
    75 weren't gave yaks
    74 hi, smile inside lesson, eh, break animals                         [oh Fluttershy…]
    72 woo-hoo, where's room, nope, definitely four either         [looking for a room?]
    71 shining hate week, brought birthday helping, goin past, flying      [is there a week for that?]
    69 filly twilight's behind shouldn't fix dragons, also called morning ahead         [LEWD]
    66 lovely mane, working safe, fault                                         [always work safe! and you will maintain lovely mane]
    65 light fact, wonderbolt pick manehattan anyway
    64 ow, c'mon, crazy sky train, moment, hurt, luck                  [who wanna ride the crazy sky train? choo-choo! ]
    63 taking truth kinda worried wonder land instead                 [yup, because the truth was that they're only a deck of cards :J ]
    62 full history: lucky royal unicorn met extra changeling, sound, spend later feeling open     [sounds like season 2 finale in a nutshell]
    59 perhaps they'll fight magical armor, gabby, line others      [yup, Gabby, line the others up, and give them the magical armor, then maybe they'll fight]
    58 obviously farm business, word trip, scared snow               [Winter Wrap Up?]
    56 cloudsdale takes top number village, goes running step, send       [...I got nothing...]
    55 sometimes plenty mess become possible, worse scary side, trees worked       [yeah, sometimes a plenty of mess is possible when you let your worse scary side to take over, but trees always work :q ]
    54 wedding school, six years longer, test against free attention star    [who's the attention star they're talking about? and why did she attend the wedding school for six years longer? :J ]
    53 poor pegasus pet shall share its problems, sugar ma'am acres, babs & angel afternoon      [something about Tank's hibernation, AJ's farm and pet-sitting? :q ]
    52 dark, ridiculous practice bein giving nothin worth small tail       [well, dark ridiculous practices never give anything worth a small tail :q  ]
    51 yup, zecora rodeo, delicious changelings, case letter eve      [Zecora rodeo with Changelings? I wanna see that!  though I must admit, season 5 finale was pretty close :q ]
    50 unless none hay high, helped control evil feelings          [yeah, getting high on hay definitely helps controlling evil feelings :J ]
    49 entire library house seriously sent quiet, worst sort hearts possibly, except forest, hah, flyer, cut beat      [sounds like Twilight's library being destroyed by "the worst possible sort of heart" (Tirek) well, at least the Everfree Forest survived, so hah! cut the beat, flyer!... ]

OK, later ones start getting long, so I'll stop here

Edited September 4, 2017 by SasQ
Minor corrections, typos, gramar & stuff... + added colours for better clarity

[DonMaguz 1,023]

12 hours ago, SasQ said:

I noticed that too, and I even played with it for a while  ... Because when you switch them around, some of them almost look like some kind of subliminal messages or secret foreshadowing heheh

Wow, you really played with the words there! Some of them are indeed amazing coincidences. 

My favorite ones from your list:

12 hours ago, SasQ said:

  650 want friends
  491 everypony best                        [true…]
  368 let's love  
  185 true problem, anypony ?                [asked Twilight in "Lesson Zero"]
  163 y'all talk, girls                     [yup, you definitely do :q ]
    97 cider trouble                            [the only trouble with cider is that there's always too few of it :q ]
    96 learned cool games                    [the games ponies play?]
    91 surprise job days                                  [yeah, I remember these days too, when I was unemployed]
    50 unless none hay high, helped control evil feelings          [yeah, getting high on hay definitely helps controlling evil feelings :J ]

The 491 used words is the best one!

With all the opinions in the fandom and great characters in the show, the general conclusion should be that, in fact, everypony is best pony! (ass odd as it sounds)