{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Managing textual data using pandas\n",
    "\n",
    "This section introduces how to prepare and manage textual data for analysis using [pandas](http://pandas.pydata.org/), a Python library for working with tabular data.\n",
    "\n",
    "After reading this section, you should know:\n",
    "\n",
    "- how to import data into a pandas DataFrame\n",
    "- how to explore data stored in a pandas DataFrame\n",
    "- how to append data to a pandas DataFrame\n",
    "- how to save the data in a pandas DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing data to pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "# Run this cell to view a YouTube video related to this topic\n",
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('UQYyCVGZbvk', height=350, width=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by importing the pandas library. \n",
    "\n",
    "Note that we can control the name of the imported module using the `as` addition to the `import` command. pandas is commonly abbreviated `pd`.\n",
    "\n",
    "This allows us to use the variable `pd` to refer to the *pandas* library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing data from a single file\n",
    "\n",
    "You must often load and prepare the data yourself, either from a single file or from multiple files.\n",
    "\n",
    "Typical formats for distributing corpora include CSV files, which stands for Comma-separated Values, and JSON, which stands for JavaScript Object Notation or simple plain text files.\n",
    "\n",
    "pandas provides plenty of functions for [reading data in various formats](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html). You can even try importing Excel sheets!\n",
    "\n",
    "The following example shows how to load a corpus from a CSV file for processing in Python using the [SFU Opinion and Comments Corpus (SOCC)](https://github.com/sfu-discourse-lab/SOCC) (Kolhatkar et al. [2020](https://doi.org/10.1007/s41701-019-00065-w)).\n",
    "\n",
    "Let's load a part of the SFU Opinion and Comments Corpus, which contains the opinion articles from [The Globe and Mail](https://www.theglobeandmail.com/), a Canadian newspaper.\n",
    "\n",
    "We can use the `read_csv()` function from pandas to read files with comma-separated values, such as the SOCC corpus.\n",
    "\n",
    "The `read_csv()` function takes a string object as input, which defines a path to the input file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the CSV file and assign the output to the variable 'socc'\n",
    "socc = pd.read_csv('data/socc_gnm_articles.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas does all the heavy lifting and returns the contents of the CSV file in a pandas [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html), which is data structure native to pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examine the type of the object stored under the variable 'socc'\n",
    "type(socc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the `head()` method of a DataFrame to check out the first five rows of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the first five rows of the DataFrame\n",
    "socc.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the DataFrame has a tabular form.\n",
    "\n",
    "The DataFrame contains several columns such as **article_id**, **title** and **article_text**, accompanied by an index for each row (**0, 1, 2, 3, 4**).\n",
    "\n",
    "The `.at[]` accessor can be used to inspect a single item in the DataFrame.\n",
    "\n",
    "Let's examine the value in the column **title** at index 123."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "socc.at[123, 'title']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "source": [
    "### Quick exercise\n",
    "\n",
    "Let's go back to the SOCC corpus stored under the variable `socc`.\n",
    "\n",
    "Who is the author (`author`) of article at index 256? \n",
    "\n",
    "How many top-level comments (`ntop_level_comments`) did the article at index 1000 receive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "### Enter your code below this line and run the cell (press Shift and Enter at the same time)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing data from multiple files\n",
    "\n",
    "Another common scenario is that you have multiple files with text data, which you want to load into *pandas*.\n",
    "\n",
    "Let's first collect the files that we want to load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the patch library\n",
    "from pathlib import Path\n",
    "\n",
    "# Create a Path object that points to the directory with data\n",
    "corpus_dir = Path('data')\n",
    "\n",
    "# Get all .txt files in the corpus directory\n",
    "corpus_files = list(corpus_dir.glob('*.txt'))\n",
    "\n",
    "# Check the corpus files\n",
    "corpus_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To accommodate our data, let's create an empty pandas DataFrame and specify its *shape* in advance, that is, the number of rows (`index`) and the names of the columns `columns`.\n",
    "\n",
    "We can determine the number of rows needed using Python's `range()` function. This function generates a list of numbers that fall within certain range, which we can use for the index of the DataFrame.\n",
    "\n",
    "In this case, we define a `range()` between `0` and the number of text files in the directory, which are stored under the variable `corpus_files`. We retrieve their number using the `len()` function, which returns the length of Python objects, if applicable.\n",
    "\n",
    "For the columns of the DataFrame, we simply create columns for filenames and their textual content by providing a list of strings to the `columns` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame and assign the result to the variable 'df'\n",
    "df = pd.DataFrame(index=range(0, len(corpus_files)), columns=['filename', 'text'])\n",
    "\n",
    "# Call the variable to inspect the output\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an empty data with rows for each file in the corpus, we can loop over the *Path* objects under `corpus_files`, read their contents and add them to the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loop over the corpus files and count each loop using enumerate()\n",
    "for i, f in enumerate(corpus_files):\n",
    "    \n",
    "    # Read the file contents\n",
    "    text = f.read_text(encoding='utf-8')\n",
    "    \n",
    "    # Get the filename from the Path object\n",
    "    filename = f.name\n",
    "\n",
    "    # Assign the text from the file to index 'i' at column 'text'\n",
    "    # using the .at accessor – note that this modifies the DataFrame\n",
    "    # \"in place\" – you don't need to assign the result into a variable\n",
    "    df.at[i, 'text'] = text\n",
    "    \n",
    "    # We then do the same to the filename\n",
    "    df.at[i, 'filename'] = filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the result by calling the variable `df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call the variable to check the output\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the DataFrame has been populated with filenames and text.\n",
    "\n",
    "Now that we know how to load data into DataFrames, we can turn towards accessing and manipulating the data that they store."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "# Run this cell to view a YouTube video related to this topic\n",
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('-zhW3YjhuzI', height=350, width=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas DataFrames can hold a lot of information, which is often organised into columns.\n",
    "\n",
    "The columns present in a DataFrame are accessible through the attribute `columns`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve the columns and their names\n",
    "socc.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas provides various methods for examining the contents of entire columns, which can be accessed just like the keys and values of a Python dictionary.\n",
    "\n",
    "The brackets `[]` can be used to access entire columns by placing the column name within the brackets as a string.\n",
    "\n",
    "Let's retrieve the contents of the column `author`, which contains author information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve the contents of the column 'author' in the DataFrame 'socc'\n",
    "socc['author']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the column `author` contains 10399 objects, as indicated by the *Length* and *dtype* properties. The numbers on the left-hand side give the index, that is, the row numbers.\n",
    "\n",
    "The columns of a pandas DataFrame consist of another object type, namely *pandas* Series. You can think of the DataFrame as an entire table, whose columns consist of Series.\n",
    "\n",
    "We can verify this by examining their type using Python's `type()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the type of 'socc' and 'socc['author']'\n",
    "type(socc), type(socc['author'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When printing out the contents of a DataFrame or Series, pandas omits everything between the first and last five rows by default. This is convenient when working with thousands of rows.\n",
    "\n",
    "This also applies to the output for methods such as `value_counts()`, which allows counting the number of unique values in a Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count unique values in the column 'author'\n",
    "socc['author'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not surprisingly, the editorial team at the *The Globe and Mail* is responsible for most of the editorials!\n",
    "\n",
    "Let's take another look at the data by visualising the result by calling the `.plot()` method for the author information column. This method calls an external library named matplotlib, which can be used to produce all kinds of plots and visualisations.\n",
    "\n",
    "More specifically, we instruct the `plot()` method to draw a bar chart by providing the string `bar` to the `kind` argument.\n",
    "\n",
    "We also use the brackets `[:10]` to limit the output to the ten most profilic authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is some Jupyter magic that allows us to render matplotlib plots in the notebooks!\n",
    "# You only need to enter this command once.\n",
    "%matplotlib inline\n",
    "\n",
    "# Count the values in the column 'author' and clip the result to top-10 before plotting.\n",
    "socc['author'].value_counts()[:10].plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For columns with numerical values, we can also use the `describe()` method to get basic descriptive statistics on the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get basic descriptive statistics for the column 'ntop_level_comments'\n",
    "socc['ntop_level_comments'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the column `ntop_level_comments` has a total of 10339 rows. \n",
    "\n",
    "The average number of comments received by an editorial is approximately 26, but this number fluctuates, as the standard deviation from the average is nearly 40.\n",
    "\n",
    " - Some editorials do not have any comments at all, as indicated by the minimum value of 0. \n",
    " - The lowest quartile shows that 25% of the data has only one comment or less (none).\n",
    " - The second quartile (50%), which is also known as the median, indicates that *half* of the data has less than 14 comments and *half* has more than 14 comments. \n",
    " - The third quartile shows that 75% of the data has 35 comments or less. \n",
    " - The most commented editorial has 1378 comments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we would like to find the articles with zero comments?\n",
    "\n",
    "We can use the DataFrame accessor `.loc` to access specific rows based on their values.\n",
    "\n",
    "The number of comments is stored in the column `ntop_level_comments`, but we also need to specify that the DataFrame stored under the variable `socc` contains the column that we wish to examine. \n",
    "\n",
    "This causes the somewhat repetitive reference to the `socc` DataFrame, which is nevertheless necessary for being explicit.\n",
    "\n",
    "We also need to provide a command for \"is equal to\". Since the single equal sign `=` is reserved for assigning variables in Python, two equal signs `==` are used for comparison.\n",
    "\n",
    "Finally, we place the value we want to evaluate against on the right-hand side of the double equal sign `==`, that is, zero for no comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get rows with no top level comments\n",
    "socc.loc[socc['ntop_level_comments'] == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns a total of 2542 rows where the value of the column `ntop_level_comments` is zero.\n",
    "\n",
    "For more complex views of the data, we can also combine multiple criteria using the `&` symbol, which is the Python operator for \"AND\".\n",
    "\n",
    "Note that individual criteria must be placed in parentheses `()` to perform the operation.\n",
    "\n",
    "Let's check if the first author in our result, Hayden King, wrote any other articles with zero comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get number of top level comments for author Hayden King\n",
    "socc.loc[(socc['ntop_level_comments'] == 0) & (socc['author'] == 'Hayden King')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "source": [
    "### Quick in-class exercise\n",
    "\n",
    "How many articles with zero top-level comments were authored by the editorial team (`GLOBE EDITORIAL`)?\n",
    "\n",
    "Write out the whole command yourself instead of copy-pasting to get an idea of the syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "### Enter your code below this line and run the cell (press Shift and Enter at the same time)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extending DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "# Run this cell to view a YouTube video related to this topic\n",
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('Yj87Gy28vqs', height=350, width=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily add information to pandas DataFrames.\n",
    "\n",
    "One common scenario could involve loading some data from an external file (such as a CSV or JSON file), performing some analyses and storing the results to the same DataFrame.\n",
    "\n",
    "We can easily add an empty column to the DataFrame. This is achieved using the column accessor `[]` and the Python datatype `None`.\n",
    "\n",
    "Let's add a new column named `comments_ratio` to the DataFrame `socc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a new column named 'comments_ratio' to the DataFrame\n",
    "socc['comments_ratio'] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the first five rows of the DataFrame\n",
    "socc.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's populate the column with some data by calculating which percentage of the comments are top-level comments, assuming that a high percentage of top-level comments indicates comments about the article, whereas a lower percentage indicates more discussion about the comments posted.\n",
    "\n",
    "To get the proportion of top-level comments out of all comments, we must divide the number of top-level comments by the number of all comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Populate the 'comments_ratio' column by calculating the ratio of top-level comments and comments\n",
    "socc['comments_ratio'] = socc['ntop_level_comments'] / socc['ncomments']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Column accessors can be used very flexibly to access and manipulate data stored in the DataFrame, as exemplified by the division."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the first five rows of the DataFrame\n",
    "socc.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the column `comments_ratio` now stores the result of our calculation!\n",
    "\n",
    "However, we should also keep in mind that some articles did not receive any comments at all: thus we would have divided zero by zero.\n",
    "\n",
    "Let's examine these cases again by retrieving articles without comments, and use the `head()` method to limit the output to the first five rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the first five comments with no top-level comments\n",
    "socc.loc[socc['ntop_level_comments'] == 0].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For these rows, the `comments_ratio` column contains values marked as `NaN` or \"not a number\".\n",
    "\n",
    "This indicates that the division was performed on these cells as well, but the result was not a number.\n",
    "\n",
    "*pandas* automatically ignores `NaN` values when performing calculations, as show by the `.describe()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get descriptive statistics for the column 'comments_ratio'\n",
    "socc['comments_ratio'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the difference in the result for the count. Only 7797 items out of 10399 were included in the calculation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we would like to do some natural language processing and store the results in the DataFrame?\n",
    "\n",
    "Let's select articles that fall within the first quartile in terms of the ratio of original comments to all comments made (`comments_ratio`) and have received more than 200 comments (`ncomments`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter the DataFrame for highly commented articles and assign the result to the variable 'talk'\n",
    "talk = socc.loc[(socc['comments_ratio'] <= 0.384) & (socc['ncomments'] >= 200)]\n",
    "\n",
    "# Call the variable to examine the output\n",
    "talk.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import spaCy, load a medium-sized language model for English and assign this model to the variable `nlp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the spaCy library\n",
    "import spacy\n",
    "\n",
    "# Note that we now load a medium-sized language model!\n",
    "nlp = spacy.load('en_core_web_md')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's limit processing to article titles and create a placeholder column to the DataFrame named `processed_title`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new column named 'processed_title'\n",
    "talk['processed_title'] = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas warns about performing this command, because `talk` is only a slice or a _view_ into the DataFrame. \n",
    "\n",
    "Assigning a new column to **only a part of the DataFrame** would cause problems by breaking the tabular structure.\n",
    "\n",
    "We can fix the situation by creating a _deep copy_ of the slice using Python's `.copy()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a deep copy of the DataFrame\n",
    "talk = talk.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try creating an empty column again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new column named 'processed_title'\n",
    "talk['processed_title'] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the first five rows of the DataFrame\n",
    "talk.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To retrieve the title for each article from the column `title`, feed it to the language model under `nlp` for processing and store the output into the column `processed_title`, we need to use the `apply()` method of a DataFrame.\n",
    "\n",
    "As the name suggests, the `apply()` method applies whatever is provided as input to the method to each row in the column.\n",
    "\n",
    "In this case, we pass the language model `nlp` to the `apply()` method, essentially retrieving the titles stored as string objects in the column `title` and \"applying\" the language model `nlp` to them.\n",
    "\n",
    "We assign the output to the DataFrame column named `processed_title`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the language model under 'nlp' to the contents of the DataFrame column 'title'\n",
    "talk['processed_title'] = talk['title'].apply(nlp)\n",
    "\n",
    "# Call the variable to check the output\n",
    "talk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the processed titles in a separate column named `processed_title`!\n",
    "\n",
    "Let's examine the first row in the DataFrame `talk`, whose index is 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the value in the column 'processed_title' at row with index 2\n",
    "talk.at[2, 'processed_title']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the type of the contained object\n",
    "type(talk.at[2, 'processed_title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the cell contains a spaCy _Doc_ object.\n",
    "\n",
    "Let's now define our own Python **function** to fetch lemmas for each noun in the title.\n",
    "\n",
    "Python functions are _defined_ using the command `def`, which is followed by the name of the function, in this case `get_nouns`. \n",
    "\n",
    "The input to the function is given in parentheses that follow the name of the function.\n",
    "\n",
    "In this case, we name a variable for the input called `nlp_text`. This is an arbitrary variable, which is needed for referring to whatever is being provided as input to the function. To put it simply, you can think of this variable as referring to any input that will be eventually provided to the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a function named 'get_nouns' that takes a single object as input.\n",
    "# We refer to this input using the variable name 'nlp_text'.\n",
    "def get_nouns(nlp_text):\n",
    "    \n",
    "    # First we make sure that the input is of correct type\n",
    "    # by using the assert command to check the input type\n",
    "    assert type(nlp_text) == spacy.tokens.doc.Doc\n",
    "    \n",
    "    # Let's set up a placeholder list for our lemmas\n",
    "    lemmas = []\n",
    "    \n",
    "    # We begin then begin looping over the Doc object\n",
    "    for token in nlp_text:\n",
    "        \n",
    "        # If the fine-grained POS tag for the token is a noun (NN)\n",
    "        if token.tag_ == 'NN':\n",
    "            \n",
    "            # Append the token lemma to the list of lemmas\n",
    "            lemmas.append(token.lemma_)\n",
    "            \n",
    "    # When the loop is complete, return the list of lemmas\n",
    "    return lemmas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined our function, we can use the function with the `.apply()` method to collect all nouns to the column `nouns`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the 'get_nouns' function to the column 'processed_title'\n",
    "talk['nouns'] = talk['processed_title'].apply(get_nouns)\n",
    "\n",
    "# Call the variable to examine the output\n",
    "talk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, an empty DataFrame column is actually not required for adding new data, because pandas creates a new column automatically through assignment, as exemplified by `talk['nouns']`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also easily extract information from DataFrames into Python's native data structures. \n",
    "\n",
    "The `tolist()` method, for instance, can be used to extract the contents of a *pandas* Series into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cast pandas Series to a list\n",
    "noun_list = talk['nouns'].tolist()\n",
    "\n",
    "# Call the variable to check the output\n",
    "noun_list[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we have now under `noun_list` is a list of lists, because each row in the `nouns` column contains a list.  \n",
    "\n",
    "Let's loop over the list and collect the items into a single list named `final_list` using the `extend()` method of a Python list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the placeholder list\n",
    "final_list = []\n",
    "\n",
    "# Loop over each list in the list of lists\n",
    "for nlist in noun_list:\n",
    "    \n",
    "    # Extend the final list with the current list\n",
    "    final_list.extend(nlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's briefly examine the first ten items in final list and then count the number of items in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the first ten items in the list\n",
    "final_list[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the length of the list\n",
    "len(final_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To plot the 10 most frequent nouns, we can cast the `final_list` into a pandas Series, count the occurrences of each lemma using `value_counts()` and plot the result using the `plot()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the list into a pandas Series, count unique nouns\n",
    "# using the value_counts() method, get the 10 most frequent\n",
    "# items [:10] and plot the result into a bar chart using the\n",
    "# plot() method and its attribute 'kind'.\n",
    "pd.Series(final_list).value_counts()[:10].plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "# Run this cell to view a YouTube video related to this topic\n",
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('yK_3QgOVXIo', height=350, width=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas DataFrames can be easily saved as *pickled* objects using the `to_pickle()` method.\n",
    "\n",
    "The `to_pickle()` method takes a string as input, which defines a path to the file in which the DataFrame should be written.\n",
    "\n",
    "Let's pickle the DataFrame with the three articles stored under `df` into a file named `pickled_df.pkl` into the directory `data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the DataFrame to disk using pickle\n",
    "df.to_pickle('data/pickled_df.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily check if the data has been saved successfully by reading the file contents using the `read_pickle()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the pickled DataFrame and assign the result to 'df_2'\n",
    "df_2 = pd.read_pickle('data/pickled_df.pkl')\n",
    "\n",
    "# Call the variable to examine the output\n",
    "df_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the DataFrames, which returns a Boolean value (True/False) for each cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compare DataFrames 'df' and 'df_2'\n",
    "df == df_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section should have given you a basic idea of the *pandas* library and how DataFrames can be used to store and manipulate textual data.\n",
    "\n",
    "In [Part III](../part_iii/01_multilingual_nlp.ipynb), you will learn more about natural language processing techniques and their applications."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}