Managing textual data using pandas#

This section introduces how to prepare and manage textual data for analysis using pandas, a Python library for working with tabular data.

After reading this section, you should know:

  • how to import data into a pandas DataFrame

  • how to explore data stored in a pandas DataFrame

  • how to append data to a pandas DataFrame

  • how to save the data in a pandas DataFrame

Importing data to pandas#

Let’s start by importing the pandas library.

Note that we can control the name of the imported module using the as addition to the import command. pandas is commonly abbreviated pd.

This allows us to use the variable pd to refer to the pandas library.

import pandas as pd

Importing data from a single file#

You must often load and prepare the data yourself, either from a single file or from multiple files.

Typical formats for distributing corpora include CSV files, which stands for Comma-separated Values, and JSON, which stands for JavaScript Object Notation or simple plain text files.

pandas provides plenty of functions for reading data in various formats. You can even try importing Excel sheets!

The following example shows how to load a corpus from a CSV file for processing in Python using the SFU Opinion and Comments Corpus (SOCC) (Kolhatkar et al. 2020).

Let’s load a part of the SFU Opinion and Comments Corpus, which contains the opinion articles from The Globe and Mail, a Canadian newspaper.

We can use the read_csv() function from pandas to read files with comma-separated values, such as the SOCC corpus.

The read_csv() function takes a string object as input, which defines a path to the input file.

# Read the CSV file and assign the output to the variable 'socc'
socc = pd.read_csv('data/socc_gnm_articles.csv')

pandas does all the heavy lifting and returns the contents of the CSV file in a pandas DataFrame, which is data structure native to pandas.

# Examine the type of the object stored under the variable 'socc'
type(socc)
pandas.core.frame.DataFrame

Let’s use the head() method of a DataFrame to check out the first five rows of the DataFrame.

# Print the first five rows of the DataFrame
socc.head(5)
article_id title article_url author published_date ncomments ntop_level_comments article_text
0 26842506 The Tories deserve another mandate - Stephen H... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2015-10-16 EDT 2187.0 1378.0 <p>All elections are choices among imperfect a...
1 26055892 Harper hysteria a sign of closed liberal minds http://www.theglobeandmail.com/opinion/harper-... Konrad Yakabuski 2015-08-24 EDT 1103.0 455.0 <p>If even a fraction of the darkness that his...
2 6929035 Too many first nations people live in a dream ... http://www.theglobeandmail.com/opinion/too-man... Jeffrey Simpson 2013-01-05 EST 1164.0 433.0 <p>Large elements of aboriginal Canada live in...
3 19047636 The Globe's editorial board endorses Tim Hudak... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2014-06-06 EDT 905.0 432.0 <p>Over four days, The Globe editorial board l...
4 11672346 Disgruntled Arab states look to strip Canada o... http://www.theglobeandmail.com/news/world/disg... Campbell Clark 2013-05-02 EDT 1129.0 411.0 <p>Growing discontent among Arab nations over ...

As you can see, the DataFrame has a tabular form.

The DataFrame contains several columns such as article_id, title and article_text, accompanied by an index for each row (0, 1, 2, 3, 4).

The .at[] accessor can be used to inspect a single item in the DataFrame.

Let’s examine the value in the column title at index 123.

socc.at[123, 'title']
"How Toronto got a 'world-class,' gold-plated, half-billion-dollar empty train"

Importing data from multiple files#

Another common scenario is that you have multiple files with text data, which you want to load into pandas.

Let’s first collect the files that we want to load.

# Import the patch library
from pathlib import Path

# Create a Path object that points to the directory with data
corpus_dir = Path('data')

# Get all .txt files in the corpus directory
corpus_files = list(corpus_dir.glob('*.txt'))

# Check the corpus files
corpus_files
[PosixPath('data/WP_1990-08-10-25A.txt'),
 PosixPath('data/NYT_1991-01-16-A15.txt'),
 PosixPath('data/WP_1991-01-17-A1B.txt')]

To accommodate our data, let’s create an empty pandas DataFrame and specify its shape in advance, that is, the number of rows (index) and the names of the columns columns.

We can determine the number of rows needed using Python’s range() function. This function generates a list of numbers that fall within certain range, which we can use for the index of the DataFrame.

In this case, we define a range() between 0 and the number of text files in the directory, which are stored under the variable corpus_files. We retrieve their number using the len() function, which returns the length of Python objects, if applicable.

For the columns of the DataFrame, we simply create columns for filenames and their textual content by providing a list of strings to the columns argument.

# Create a DataFrame and assign the result to the variable 'df'
df = pd.DataFrame(index=range(0, len(corpus_files)), columns=['filename', 'text'])

# Call the variable to inspect the output
df
filename text
0 NaN NaN
1 NaN NaN
2 NaN NaN

Now that we have an empty data with rows for each file in the corpus, we can loop over the Path objects under corpus_files, read their contents and add them to the DataFrame.

# Loop over the corpus files and count each loop using enumerate()
for i, f in enumerate(corpus_files):
    
    # Read the file contents
    text = f.read_text(encoding='utf-8')
    
    # Get the filename from the Path object
    filename = f.name

    # Assign the text from the file to index 'i' at column 'text'
    # using the .at accessor – note that this modifies the DataFrame
    # "in place" – you don't need to assign the result into a variable
    df.at[i, 'text'] = text
    
    # We then do the same to the filename
    df.at[i, 'filename'] = filename

Let’s check the result by calling the variable df.

# Call the variable to check the output
df
filename text
0 WP_1990-08-10-25A.txt *We Don’t Stand for Bullies': Diverse Voices ...
1 NYT_1991-01-16-A15.txt U.S. TAKING STEPS TO CURB TERRORISM: F.B.I. I...
2 WP_1991-01-17-A1B.txt U.S., Allies Launch Massive Air War Against T...

As you can see, the DataFrame has been populated with filenames and text.

Now that we know how to load data into DataFrames, we can turn towards accessing and manipulating the data that they store.

Examining DataFrames#

pandas DataFrames can hold a lot of information, which is often organised into columns.

The columns present in a DataFrame are accessible through the attribute columns.

# Retrieve the columns and their names
socc.columns
Index(['article_id', 'title', 'article_url', 'author', 'published_date',
       'ncomments', 'ntop_level_comments', 'article_text'],
      dtype='object')

pandas provides various methods for examining the contents of entire columns, which can be accessed just like the keys and values of a Python dictionary.

The brackets [] can be used to access entire columns by placing the column name within the brackets as a string.

Let’s retrieve the contents of the column author, which contains author information.

# Retrieve the contents of the column 'author' in the DataFrame 'socc'
socc['author']
0         GLOBE EDITORIAL
1        Konrad Yakabuski
2         Jeffrey Simpson
3         GLOBE EDITORIAL
4          Campbell Clark
               ...       
10334     GLOBE EDITORIAL
10335     GLOBE EDITORIAL
10336     GLOBE EDITORIAL
10337      Adam Radwanski
10338     GLOBE EDITORIAL
Name: author, Length: 10339, dtype: object

As you can see, the column author contains 10399 objects, as indicated by the Length and dtype properties. The numbers on the left-hand side give the index, that is, the row numbers.

The columns of a pandas DataFrame consist of another object type, namely pandas Series. You can think of the DataFrame as an entire table, whose columns consist of Series.

We can verify this by examining their type using Python’s type() function.

# Check the type of 'socc' and 'socc['author']'
type(socc), type(socc['author'])
(pandas.core.frame.DataFrame, pandas.core.series.Series)

When printing out the contents of a DataFrame or Series, pandas omits everything between the first and last five rows by default. This is convenient when working with thousands of rows.

This also applies to the output for methods such as value_counts(), which allows counting the number of unique values in a Series.

# Count unique values in the column 'author'
socc['author'].value_counts()
GLOBE EDITORIAL                   2712
Jeffrey Simpson                    649
Margaret Wente                     547
Konrad Yakabuski                   404
Gary Mason                         365
                                  ... 
MARK MATTSON AND KRYSTYN TULLY       1
KEVIN LACEY                          1
Adrian Morrow                        1
Jessica Scott-Reid                   1
Kenneth Oppel                        1
Name: author, Length: 1896, dtype: int64

Not surprisingly, the editorial team at the The Globe and Mail is responsible for most of the editorials!

Let’s take another look at the data by visualising the result by calling the .plot() method for the author information column. This method calls an external library named matplotlib, which can be used to produce all kinds of plots and visualisations.

More specifically, we instruct the plot() method to draw a bar chart by providing the string bar to the kind argument.

We also use the brackets [:10] to limit the output to the ten most profilic authors.

# This is some Jupyter magic that allows us to render matplotlib plots in the notebooks!
# You only need to enter this command once.
%matplotlib inline

# Count the values in the column 'author' and clip the result to top-10 before plotting.
socc['author'].value_counts()[:10].plot(kind='bar')
<AxesSubplot:>
../../_images/1dfb53e275799f1a8468e434425b04857b9f8fec53350dbbbc4c66c67a578f17.png

For columns with numerical values, we can also use the describe() method to get basic descriptive statistics on the data.

# Get basic descriptive statistics for the column 'ntop_level_comments'
socc['ntop_level_comments'].describe()
count    10339.000000
mean        26.384273
std         39.786923
min          0.000000
25%          1.000000
50%         14.000000
75%         35.000000
max       1378.000000
Name: ntop_level_comments, dtype: float64

As we can see, the column ntop_level_comments has a total of 10339 rows.

The average number of comments received by an editorial is approximately 26, but this number fluctuates, as the standard deviation from the average is nearly 40.

  • Some editorials do not have any comments at all, as indicated by the minimum value of 0.

  • The lowest quartile shows that 25% of the data has only one comment or less (none).

  • The second quartile (50%), which is also known as the median, indicates that half of the data has less than 14 comments and half has more than 14 comments.

  • The third quartile shows that 75% of the data has 35 comments or less.

  • The most commented editorial has 1378 comments.

What if we would like to find the articles with zero comments?

We can use the DataFrame accessor .loc to access specific rows based on their values.

The number of comments is stored in the column ntop_level_comments, but we also need to specify that the DataFrame stored under the variable socc contains the column that we wish to examine.

This causes the somewhat repetitive reference to the socc DataFrame, which is nevertheless necessary for being explicit.

We also need to provide a command for “is equal to”. Since the single equal sign = is reserved for assigning variables in Python, two equal signs == are used for comparison.

Finally, we place the value we want to evaluate against on the right-hand side of the double equal sign ==, that is, zero for no comments.

# Get rows with no top level comments
socc.loc[socc['ntop_level_comments'] == 0]
article_id title article_url author published_date ncomments ntop_level_comments article_text
7797 33441604 Joseph Boyden, where are you from? http://www.theglobeandmail.com/opinion/joseph-... Hayden King 2016-12-28 EST 0.0 0.0 <p>Hayden King teaches in the School of Public...
7798 33316285 Globe editorial: Rejoice! Congress just gave t... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2016-12-13 EST 0.0 0.0 <p>The United States may have just elected a p...
7799 33009790 Police and La Presse: Warrants not warranted http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2016-11-23 EST 0.0 0.0 <p>The discovery that the Montreal Police obta...
7800 32970624 The Galloway affair: Salem comes to UBC http://www.theglobeandmail.com/opinion/the-gal... Margaret Wente 2016-11-22 EST 0.0 0.0 <p>I have a question about the Steven Galloway...
7801 32927142 Justice delayed: the law of unintended consequ... http://www.theglobeandmail.com/opinion/unreaso... BENJAMIN PERRIN 2016-11-19 EST 0.0 0.0 <p>Benjamin Perrin is a law professor at the U...
... ... ... ... ... ... ... ... ...
10334 533784 WTO action on China's rare-earth quotas makes ... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2012-03-14 EDT 0.0 0.0 <p>The confusingly named substances known as '...
10335 533594 A customer-friendly Finance Department http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2012-03-13 EDT 0.0 0.0 <p>Of the many things that frustrate the retai...
10336 533508 Video raises questions about Nik Zoricic's 'fr... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2012-03-12 EDT 0.0 0.0 <p>Officials and fans are mourning the death o...
10337 533504 McGuinty can't afford misgivings about gaming http://www.theglobeandmail.com/news/politics/m... Adam Radwanski 2012-03-12 EDT 0.0 0.0 <p>Unlike so many of the other measures that m...
10338 533471 In Russia, Canada should look for investment, ... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2012-03-12 EDT 0.0 0.0 <p>As if in swift response to Prime Minister V...

2542 rows × 8 columns

This returns a total of 2542 rows where the value of the column ntop_level_comments is zero.

For more complex views of the data, we can also combine multiple criteria using the & symbol, which is the Python operator for “AND”.

Note that individual criteria must be placed in parentheses () to perform the operation.

Let’s check if the first author in our result, Hayden King, wrote any other articles with zero comments.

# Get number of top level comments for author Hayden King
socc.loc[(socc['ntop_level_comments'] == 0) & (socc['author'] == 'Hayden King')]
article_id title article_url author published_date ncomments ntop_level_comments article_text
7797 33441604 Joseph Boyden, where are you from? http://www.theglobeandmail.com/opinion/joseph-... Hayden King 2016-12-28 EST 0.0 0.0 <p>Hayden King teaches in the School of Public...

Extending DataFrames#

You can easily add information to pandas DataFrames.

One common scenario could involve loading some data from an external file (such as a CSV or JSON file), performing some analyses and storing the results to the same DataFrame.

We can easily add an empty column to the DataFrame. This is achieved using the column accessor [] and the Python datatype None.

Let’s add a new column named comments_ratio to the DataFrame socc.

# Add a new column named 'comments_ratio' to the DataFrame
socc['comments_ratio'] = None
# Print out the first five rows of the DataFrame
socc.head(5)
article_id title article_url author published_date ncomments ntop_level_comments article_text comments_ratio
0 26842506 The Tories deserve another mandate - Stephen H... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2015-10-16 EDT 2187.0 1378.0 <p>All elections are choices among imperfect a... None
1 26055892 Harper hysteria a sign of closed liberal minds http://www.theglobeandmail.com/opinion/harper-... Konrad Yakabuski 2015-08-24 EDT 1103.0 455.0 <p>If even a fraction of the darkness that his... None
2 6929035 Too many first nations people live in a dream ... http://www.theglobeandmail.com/opinion/too-man... Jeffrey Simpson 2013-01-05 EST 1164.0 433.0 <p>Large elements of aboriginal Canada live in... None
3 19047636 The Globe's editorial board endorses Tim Hudak... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2014-06-06 EDT 905.0 432.0 <p>Over four days, The Globe editorial board l... None
4 11672346 Disgruntled Arab states look to strip Canada o... http://www.theglobeandmail.com/news/world/disg... Campbell Clark 2013-05-02 EDT 1129.0 411.0 <p>Growing discontent among Arab nations over ... None

Let’s populate the column with some data by calculating which percentage of the comments are top-level comments, assuming that a high percentage of top-level comments indicates comments about the article, whereas a lower percentage indicates more discussion about the comments posted.

To get the proportion of top-level comments out of all comments, we must divide the number of top-level comments by the number of all comments.

# Populate the 'comments_ratio' column by calculating the ratio of top-level comments and comments
socc['comments_ratio'] = socc['ntop_level_comments'] / socc['ncomments']

Column accessors can be used very flexibly to access and manipulate data stored in the DataFrame, as exemplified by the division.

# Print out the first five rows of the DataFrame
socc.head(5)
article_id title article_url author published_date ncomments ntop_level_comments article_text comments_ratio
0 26842506 The Tories deserve another mandate - Stephen H... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2015-10-16 EDT 2187.0 1378.0 <p>All elections are choices among imperfect a... 0.630087
1 26055892 Harper hysteria a sign of closed liberal minds http://www.theglobeandmail.com/opinion/harper-... Konrad Yakabuski 2015-08-24 EDT 1103.0 455.0 <p>If even a fraction of the darkness that his... 0.412511
2 6929035 Too many first nations people live in a dream ... http://www.theglobeandmail.com/opinion/too-man... Jeffrey Simpson 2013-01-05 EST 1164.0 433.0 <p>Large elements of aboriginal Canada live in... 0.371993
3 19047636 The Globe's editorial board endorses Tim Hudak... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2014-06-06 EDT 905.0 432.0 <p>Over four days, The Globe editorial board l... 0.477348
4 11672346 Disgruntled Arab states look to strip Canada o... http://www.theglobeandmail.com/news/world/disg... Campbell Clark 2013-05-02 EDT 1129.0 411.0 <p>Growing discontent among Arab nations over ... 0.364039

As you can see, the column comments_ratio now stores the result of our calculation!

However, we should also keep in mind that some articles did not receive any comments at all: thus we would have divided zero by zero.

Let’s examine these cases again by retrieving articles without comments, and use the head() method to limit the output to the first five rows.

# Print out the first five comments with no top-level comments
socc.loc[socc['ntop_level_comments'] == 0].head(5)
article_id title article_url author published_date ncomments ntop_level_comments article_text comments_ratio
7797 33441604 Joseph Boyden, where are you from? http://www.theglobeandmail.com/opinion/joseph-... Hayden King 2016-12-28 EST 0.0 0.0 <p>Hayden King teaches in the School of Public... NaN
7798 33316285 Globe editorial: Rejoice! Congress just gave t... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2016-12-13 EST 0.0 0.0 <p>The United States may have just elected a p... NaN
7799 33009790 Police and La Presse: Warrants not warranted http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2016-11-23 EST 0.0 0.0 <p>The discovery that the Montreal Police obta... NaN
7800 32970624 The Galloway affair: Salem comes to UBC http://www.theglobeandmail.com/opinion/the-gal... Margaret Wente 2016-11-22 EST 0.0 0.0 <p>I have a question about the Steven Galloway... NaN
7801 32927142 Justice delayed: the law of unintended consequ... http://www.theglobeandmail.com/opinion/unreaso... BENJAMIN PERRIN 2016-11-19 EST 0.0 0.0 <p>Benjamin Perrin is a law professor at the U... NaN

For these rows, the comments_ratio column contains values marked as NaN or “not a number”.

This indicates that the division was performed on these cells as well, but the result was not a number.

pandas automatically ignores NaN values when performing calculations, as show by the .describe() method.

# Get descriptive statistics for the column 'comments_ratio'
socc['comments_ratio'].describe()
count    7797.000000
mean        0.537057
std         0.205398
min         0.083333
25%         0.384615
50%         0.485714
75%         0.647059
max         1.000000
Name: comments_ratio, dtype: float64

Note the difference in the result for the count. Only 7797 items out of 10399 were included in the calculation.

What if we would like to do some natural language processing and store the results in the DataFrame?

Let’s select articles that fall within the first quartile in terms of the ratio of original comments to all comments made (comments_ratio) and have received more than 200 comments (ncomments).

# Filter the DataFrame for highly commented articles and assign the result to the variable 'talk'
talk = socc.loc[(socc['comments_ratio'] <= 0.384) & (socc['ncomments'] >= 200)]

# Call the variable to examine the output
talk.head(5)
article_id title article_url author published_date ncomments ntop_level_comments article_text comments_ratio
2 6929035 Too many first nations people live in a dream ... http://www.theglobeandmail.com/opinion/too-man... Jeffrey Simpson 2013-01-05 EST 1164.0 433.0 <p>Large elements of aboriginal Canada live in... 0.371993
4 11672346 Disgruntled Arab states look to strip Canada o... http://www.theglobeandmail.com/news/world/disg... Campbell Clark 2013-05-02 EDT 1129.0 411.0 <p>Growing discontent among Arab nations over ... 0.364039
5 26691065 Fifty years in Canada, and now I feel like a s... http://www.theglobeandmail.com/opinion/fifty-y... SHEEMA KHAN 2015-10-07 EDT 1142.0 376.0 <p>'Too broken to write,' I told my editor, af... 0.329247
6 25731634 I'm Canadian - and I should have a right to vote http://www.theglobeandmail.com/opinion/im-cana... Donald Sutherland 2015-07-28 EDT 1021.0 348.0 <p>My name is Donald Sutherland. My wife's nam... 0.340842
8 13647608 A nation of $100,000 firefighters http://www.theglobeandmail.com/opinion/a-natio... Margaret Wente 2013-08-08 EDT 1102.0 338.0 <p>Everyone loves firefighters. They save live... 0.306715

Let’s import spaCy, load a medium-sized language model for English and assign this model to the variable nlp.

# Import the spaCy library
import spacy

# Note that we now load a medium-sized language model!
nlp = spacy.load('en_core_web_md')

Let’s limit processing to article titles and create a placeholder column to the DataFrame named processed_title.

# Create a new column named 'processed_title'
talk['processed_title'] = None
/var/folders/qw/3f1xq85n6vl73ksgmtslw2_mzhdr7p/T/ipykernel_73735/1354777276.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  talk['processed_title'] = None

pandas warns about performing this command, because talk is only a slice or a view into the DataFrame.

Assigning a new column to only a part of the DataFrame would cause problems by breaking the tabular structure.

We can fix the situation by creating a deep copy of the slice using Python’s .copy() method.

# Create a deep copy of the DataFrame
talk = talk.copy()

Let’s try creating an empty column again.

# Create a new column named 'processed_title'
talk['processed_title'] = None
# Print out the first five rows of the DataFrame
talk.head(5)
article_id title article_url author published_date ncomments ntop_level_comments article_text comments_ratio processed_title
2 6929035 Too many first nations people live in a dream ... http://www.theglobeandmail.com/opinion/too-man... Jeffrey Simpson 2013-01-05 EST 1164.0 433.0 <p>Large elements of aboriginal Canada live in... 0.371993 None
4 11672346 Disgruntled Arab states look to strip Canada o... http://www.theglobeandmail.com/news/world/disg... Campbell Clark 2013-05-02 EDT 1129.0 411.0 <p>Growing discontent among Arab nations over ... 0.364039 None
5 26691065 Fifty years in Canada, and now I feel like a s... http://www.theglobeandmail.com/opinion/fifty-y... SHEEMA KHAN 2015-10-07 EDT 1142.0 376.0 <p>'Too broken to write,' I told my editor, af... 0.329247 None
6 25731634 I'm Canadian - and I should have a right to vote http://www.theglobeandmail.com/opinion/im-cana... Donald Sutherland 2015-07-28 EDT 1021.0 348.0 <p>My name is Donald Sutherland. My wife's nam... 0.340842 None
8 13647608 A nation of $100,000 firefighters http://www.theglobeandmail.com/opinion/a-natio... Margaret Wente 2013-08-08 EDT 1102.0 338.0 <p>Everyone loves firefighters. They save live... 0.306715 None

To retrieve the title for each article from the column title, feed it to the language model under nlp for processing and store the output into the column processed_title, we need to use the apply() method of a DataFrame.

As the name suggests, the apply() method applies whatever is provided as input to the method to each row in the column.

In this case, we pass the language model nlp to the apply() method, essentially retrieving the titles stored as string objects in the column title and “applying” the language model nlp to them.

We assign the output to the DataFrame column named processed_title.

# Apply the language model under 'nlp' to the contents of the DataFrame column 'title'
talk['processed_title'] = talk['title'].apply(nlp)

# Call the variable to check the output
talk
article_id title article_url author published_date ncomments ntop_level_comments article_text comments_ratio processed_title
2 6929035 Too many first nations people live in a dream ... http://www.theglobeandmail.com/opinion/too-man... Jeffrey Simpson 2013-01-05 EST 1164.0 433.0 <p>Large elements of aboriginal Canada live in... 0.371993 (Too, many, first, nations, people, live, in, ...
4 11672346 Disgruntled Arab states look to strip Canada o... http://www.theglobeandmail.com/news/world/disg... Campbell Clark 2013-05-02 EDT 1129.0 411.0 <p>Growing discontent among Arab nations over ... 0.364039 (Disgruntled, Arab, states, look, to, strip, C...
5 26691065 Fifty years in Canada, and now I feel like a s... http://www.theglobeandmail.com/opinion/fifty-y... SHEEMA KHAN 2015-10-07 EDT 1142.0 376.0 <p>'Too broken to write,' I told my editor, af... 0.329247 (Fifty, years, in, Canada, ,, and, now, I, fee...
6 25731634 I'm Canadian - and I should have a right to vote http://www.theglobeandmail.com/opinion/im-cana... Donald Sutherland 2015-07-28 EDT 1021.0 348.0 <p>My name is Donald Sutherland. My wife's nam... 0.340842 (I, 'm, Canadian, -, and, I, should, have, a, ...
8 13647608 A nation of $100,000 firefighters http://www.theglobeandmail.com/opinion/a-natio... Margaret Wente 2013-08-08 EDT 1102.0 338.0 <p>Everyone loves firefighters. They save live... 0.306715 (A, nation, of, $, 100,000, firefighters)
... ... ... ... ... ... ... ... ... ... ...
1694 30474884 A dangerous moment in history: Can the politic... http://www.theglobeandmail.com/opinion/can-the... Konrad Yakabuski 2016-06-16 EDT 239.0 50.0 <p>As anyone trying to maintain perspective wh... 0.209205 (A, dangerous, moment, in, history, :, Can, th...
1735 32088785 Clinton shines in first debate, and not just i... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2016-09-27 EDT 232.0 49.0 <p>For those who wondered whether Hillary Clin... 0.211207 (Clinton, shines, in, first, debate, ,, and, n...
2213 30508530 U.S. gun control: Don't look for logic after O... http://www.theglobeandmail.com/opinion/us-gun-... Konrad Yakabuski 2016-06-20 EDT 243.0 40.0 <p>The script is by now tediously formulaic. A... 0.164609 (U.S., gun, control, :, Do, n't, look, for, lo...
2301 31605288 Let's make sure Ontario's sex-ed curriculum is... http://www.theglobeandmail.com/opinion/lets-ma... DEBRA SOH 2016-08-30 EDT 239.0 39.0 <p>Debra W. Soh is a sex writer and sexual neu... 0.163180 (Let, 's, make, sure, Ontario, 's, sex, -, ed,...
2302 24363093 Dad rules when sex ed collides with religion http://www.theglobeandmail.com/opinion/dad-rul... MICHAEL ADAMS 2015-05-11 EDT 222.0 39.0 <p>Michael Adams is founder and president of t... 0.175676 (Dad, rules, when, sex, ed, collides, with, re...

519 rows × 10 columns

We now have the processed titles in a separate column named processed_title!

Let’s examine the first row in the DataFrame talk, whose index is 2.

# Get the value in the column 'processed_title' at row with index 2
talk.at[2, 'processed_title']
Too many first nations people live in a dream palace
# Check the type of the contained object
type(talk.at[2, 'processed_title'])
spacy.tokens.doc.Doc

As you can see, the cell contains a spaCy Doc object.

Let’s now define our own Python function to fetch lemmas for each noun in the title.

Python functions are defined using the command def, which is followed by the name of the function, in this case get_nouns.

The input to the function is given in parentheses that follow the name of the function.

In this case, we name a variable for the input called nlp_text. This is an arbitrary variable, which is needed for referring to whatever is being provided as input to the function. To put it simply, you can think of this variable as referring to any input that will be eventually provided to the function.

# Define a function named 'get_nouns' that takes a single object as input.
# We refer to this input using the variable name 'nlp_text'.
def get_nouns(nlp_text):
    
    # First we make sure that the input is of correct type
    # by using the assert command to check the input type
    assert type(nlp_text) == spacy.tokens.doc.Doc
    
    # Let's set up a placeholder list for our lemmas
    lemmas = []
    
    # We begin then begin looping over the Doc object
    for token in nlp_text:
        
        # If the fine-grained POS tag for the token is a noun (NN)
        if token.tag_ == 'NN':
            
            # Append the token lemma to the list of lemmas
            lemmas.append(token.lemma_)
            
    # When the loop is complete, return the list of lemmas
    return lemmas

Now that we have defined our function, we can use the function with the .apply() method to collect all nouns to the column nouns.

# Apply the 'get_nouns' function to the column 'processed_title'
talk['nouns'] = talk['processed_title'].apply(get_nouns)

# Call the variable to examine the output
talk
article_id title article_url author published_date ncomments ntop_level_comments article_text comments_ratio processed_title nouns
2 6929035 Too many first nations people live in a dream ... http://www.theglobeandmail.com/opinion/too-man... Jeffrey Simpson 2013-01-05 EST 1164.0 433.0 <p>Large elements of aboriginal Canada live in... 0.371993 (Too, many, first, nations, people, live, in, ... [dream, palace]
4 11672346 Disgruntled Arab states look to strip Canada o... http://www.theglobeandmail.com/news/world/disg... Campbell Clark 2013-05-02 EDT 1129.0 411.0 <p>Growing discontent among Arab nations over ... 0.364039 (Disgruntled, Arab, states, look, to, strip, C... [agency]
5 26691065 Fifty years in Canada, and now I feel like a s... http://www.theglobeandmail.com/opinion/fifty-y... SHEEMA KHAN 2015-10-07 EDT 1142.0 376.0 <p>'Too broken to write,' I told my editor, af... 0.329247 (Fifty, years, in, Canada, ,, and, now, I, fee... [class, citizen]
6 25731634 I'm Canadian - and I should have a right to vote http://www.theglobeandmail.com/opinion/im-cana... Donald Sutherland 2015-07-28 EDT 1021.0 348.0 <p>My name is Donald Sutherland. My wife's nam... 0.340842 (I, 'm, Canadian, -, and, I, should, have, a, ... [right]
8 13647608 A nation of $100,000 firefighters http://www.theglobeandmail.com/opinion/a-natio... Margaret Wente 2013-08-08 EDT 1102.0 338.0 <p>Everyone loves firefighters. They save live... 0.306715 (A, nation, of, $, 100,000, firefighters) [nation]
... ... ... ... ... ... ... ... ... ... ... ...
1694 30474884 A dangerous moment in history: Can the politic... http://www.theglobeandmail.com/opinion/can-the... Konrad Yakabuski 2016-06-16 EDT 239.0 50.0 <p>As anyone trying to maintain perspective wh... 0.209205 (A, dangerous, moment, in, history, :, Can, th... [moment, history, centre, hold]
1735 32088785 Clinton shines in first debate, and not just i... http://www.theglobeandmail.com/opinion/editori... GLOBE EDITORIAL 2016-09-27 EDT 232.0 49.0 <p>For those who wondered whether Hillary Clin... 0.211207 (Clinton, shines, in, first, debate, ,, and, n... [debate, comparison]
2213 30508530 U.S. gun control: Don't look for logic after O... http://www.theglobeandmail.com/opinion/us-gun-... Konrad Yakabuski 2016-06-20 EDT 243.0 40.0 <p>The script is by now tediously formulaic. A... 0.164609 (U.S., gun, control, :, Do, n't, look, for, lo... [gun, control, logic]
2301 31605288 Let's make sure Ontario's sex-ed curriculum is... http://www.theglobeandmail.com/opinion/lets-ma... DEBRA SOH 2016-08-30 EDT 239.0 39.0 <p>Debra W. Soh is a sex writer and sexual neu... 0.163180 (Let, 's, make, sure, Ontario, 's, sex, -, ed,... [sex, ed, curriculum]
2302 24363093 Dad rules when sex ed collides with religion http://www.theglobeandmail.com/opinion/dad-rul... MICHAEL ADAMS 2015-05-11 EDT 222.0 39.0 <p>Michael Adams is founder and president of t... 0.175676 (Dad, rules, when, sex, ed, collides, with, re... [sex, religion]

519 rows × 11 columns

As you can see, an empty DataFrame column is actually not required for adding new data, because pandas creates a new column automatically through assignment, as exemplified by talk['nouns'].

We can also easily extract information from DataFrames into Python’s native data structures.

The tolist() method, for instance, can be used to extract the contents of a pandas Series into a list.

# Cast pandas Series to a list
noun_list = talk['nouns'].tolist()

# Call the variable to check the output
noun_list[:10]
[['dream', 'palace'],
 ['agency'],
 ['class', 'citizen'],
 ['right'],
 ['nation'],
 [],
 ['reform'],
 ['leader', 'parade'],
 ['pm'],
 ['government', 'monopoly']]

What we have now under noun_list is a list of lists, because each row in the nouns column contains a list.

Let’s loop over the list and collect the items into a single list named final_list using the extend() method of a Python list.

# Set up the placeholder list
final_list = []

# Loop over each list in the list of lists
for nlist in noun_list:
    
    # Extend the final list with the current list
    final_list.extend(nlist)

Let’s briefly examine the first ten items in final list and then count the number of items in the list.

# Print the first ten items in the list
final_list[:10]
['dream',
 'palace',
 'agency',
 'class',
 'citizen',
 'right',
 'nation',
 'reform',
 'leader',
 'parade']
# Check the length of the list
len(final_list)
884

To plot the 10 most frequent nouns, we can cast the final_list into a pandas Series, count the occurrences of each lemma using value_counts() and plot the result using the plot() method.

# Convert the list into a pandas Series, count unique nouns
# using the value_counts() method, get the 10 most frequent
# items [:10] and plot the result into a bar chart using the
# plot() method and its attribute 'kind'.
pd.Series(final_list).value_counts()[:10].plot(kind='bar')
<AxesSubplot:>
../../_images/2a8978ec540c381f614c96fddd5510876a34299ee3c97463981e6940873fe7fc.png

Saving DataFrames#

pandas DataFrames can be easily saved as pickled objects using the to_pickle() method.

The to_pickle() method takes a string as input, which defines a path to the file in which the DataFrame should be written.

Let’s pickle the DataFrame with the three articles stored under df into a file named pickled_df.pkl into the directory data.

# Write the DataFrame to disk using pickle
df.to_pickle('data/pickled_df.pkl')

We can easily check if the data has been saved successfully by reading the file contents using the read_pickle() method.

# Read the pickled DataFrame and assign the result to 'df_2'
df_2 = pd.read_pickle('data/pickled_df.pkl')

# Call the variable to examine the output
df_2
filename text
0 WP_1990-08-10-25A.txt *We Don’t Stand for Bullies': Diverse Voices ...
1 NYT_1991-01-16-A15.txt U.S. TAKING STEPS TO CURB TERRORISM: F.B.I. I...
2 WP_1991-01-17-A1B.txt U.S., Allies Launch Massive Air War Against T...

Let’s compare the DataFrames, which returns a Boolean value (True/False) for each cell.

# Compare DataFrames 'df' and 'df_2'
df == df_2
filename text
0 True True
1 True True
2 True True

This section should have given you a basic idea of the pandas library and how DataFrames can be used to store and manipulate textual data.

In Part III, you will learn more about natural language processing techniques and their applications.