Sokratis Papadopoulos data | urban | policy

Expression of emotions in literature through big data

In this post, inspired by this very interesting work, I process Google 1-grams to analyze patterns in English and German 20th century literature emotions.

I try to replicate the paper’s results and extend the method to other languages. I am showing how to process large-scale text data using functional programming and without loading entire files in memory.

Let’s get started!

First, we pull the English emotion-related lexicons from the WordNet-Affect project. Then, we call the Google Translate API to translate the lexicons in German.

translator = Translator()

listOfEmotions = ['anger', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'normalize']

# Load all WordNetAffect emotion lists and store them in a dictionary from kylehg's github sentiment 
# analysis repo:
for emotion in listOfEmotions:
    if emotion not in 'normalize':
        words = []
        source = urllib2.urlopen(r""
                                + emotion +".txt") 
        for line in source:
        # store english words in dictionary
        words = list(itertools.chain(*words))
        emotionD_eng[emotion] = words      
        # translate and store german words
            words_ger = map(lambda x: unicodedata.normalize('NFD', translator.translate(x, dest='de').text).encode('ascii', 'ignore').lower(), words)
        except ValueError:
        emotionD_ger[emotion] = words_ger          
        emotionD_eng[emotion] = [[u'the']]
        emotionD_ger[emotion] = [[u'das']]

Now we have two dictionaries, an English and a German, including the words associated with all six emotions we analyze. Let’s download the 1-grams included in all the books published during the 20th century.

# Create directories to store the .csv files
# Create directories to store the processed .pkl files

# Download Google 1-grams
for num in range(10):
    # English 1-grams
    url = ""+str(num)+""
    filename = url.split("/")[-1]
    with open(filename, "wb") as f:
        r = requests.get(url)
    # German 1-grams
    url = ""+str(num)+""
    filename = url.split("/")[-1]
    with open(filename, "wb") as f:
        r = requests.get(url)
# Unzip data and move to directory
for fileName in  os.listdir('.'):
    if '.zip' in fileName:
        zip_ref = zipfile.ZipFile(fileName, 'r')
        for csvFile in glob.glob('*.csv'):
            if '-eng-' in csvFile:
                shutil.move(csvFile, "eng/"+csvFile)
                shutil.move(csvFile, "ger/"+csvFile)

Next I write a couple of map and filter functions to process our data efficiently. Along with the emotional words I count the word “the” (ger: “das”) occurrences as a normalization factor.

# 'the' is the normalization factor
listOfEmotions = ['anger', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'the']

# load pickled emotion words dictionaries
emotionD_eng, emotionD_ger = cPickle.load(open('wordNetdicts.pkl', 'r'))

# Aggregate emotional words by year
def mapper(row):
    return unicodedata.normalize('NFD', row['word'].decode('utf-8')), int(row['year']), int(row['counts'])
def mapper2(row):
    return (row[1], row[2])

# Process
for lang in ['eng', 'ger']:
    if lang == 'eng':
    	# Due to change in encoding in compute environment
        emotionD_eng['the'] = [[u'the']]
        emotionDic = emotionD_eng
        emotionD_ger['the'] = [[u'das']]
        emotionDic = emotionD_ger
    for emotion in listOfEmotions:
        def filterer(filt):
            # select 20th centrury emotional words
            if (filt[1]<=2000)&(filt[1]>=1900)&(filt[0] in emotionDic[emotion]):
                return filt
        counts = []
        counter = 1
        os.chdir(os.getcwd() + '/' + lang)
        for csvFile in glob.glob('*.csv'):
            with open(csvFile, 'rb') as csvfile:
                reader = csv.DictReader(csvfile, fieldnames=['word', 'year', 'counts', 'page', 'volume'],
                                        delimiter='\t', quoting=csv.QUOTE_NONE)
                output = map(mapper2, filter(filterer,map(mapper, reader)))

            counts = counts+output
            print "Emotion:" + emotion + " -- Processed " + str(counter) + "/" + str(len(glob.glob('*.csv'))) + " files"

        dictionary = dict()
        for (year, val) in counts: 
            dictionary[year] = dictionary.get(year, 0) + val  # return the value for that key or return default 0 (and create key)

        data_aggregated = [(key, val) for (key, val) in dictionary.iteritems()]
        with open('pickledEmotions/'+ lang + '/' + emotion + '_'+ str.upper(lang) +'.pkl', 'w') as f:  
            pickle.dump([data_aggregated], f)

        aggrEmotDic[emotion] = data_aggregated

Great, we were able to extract all the emotion-related words from almost 20GB of data! Now we can compute means and z-scores (see this notebook for details) and visualize the results.

We can see evidence that emotion in literature reflects the general mood associated with historic events. Look, for instance, the drop in positive emotions during WW2. We also notice that positive vs negative emotions fluctuate in English literature, unlike German which is quite steady over time with few spikes.

Another interesting finding is that the overall emotion-related word usage drops steadily through time for English literature. This is not the case for German literature, which shows increasing emotion usage, especially after the 1950s.

Finally, I plot a breakdown of z-scores for each emotion. A couple of interesting things here. Look at the spike in the fear-related words in German literature right after the end of WW2. Also, see how anger-related words increase in English literature during this period.


In this post I analyzed trends in the use of emotion-related words in literature. I show how to process large-scale text data in a computationally efficient manner.

Plotting the results reveals interesting associations between emotion and historical events, as well as differential patterns over time between English and German literature. You can find the full code in this GitHub repository.