Part II: Working with Text in Python

Part II introduces you to working with text using the Python programming language.

After reading through this section, you should be able to:

• understand how computers process and store text

• how to load text into Python for processing and save the results

• know basic tasks in natural language processing and the kinds of annotations they produce

• how to evaluate the performance of natural language processing algorithms

• how to store texts and prepare them for further analysis

Tutorials on YouTube

Table of Contents

• Manipulating text using Python
  • Computers and text
    • What is rich text?
    • What is plain text?
    • What is structured text?
    • Why does this matter?
  • Text encoding
    • ASCII
    • Unicode
  • Loading plain text files into Python
  • Manipulating text
• Manipulating text at scale
  • Regular expressions
  • Processing multiple files
• Processing texts using spaCy
  • Getting started
    • What is a language model?
    • How are language models trained?
  • Performing basic NLP tasks using spaCy
    • Tokenization
    • Part-of-speech tagging
    • Morphological analysis
    • Syntactic parsing
    • Sentence segmentation
    • Lemmatization
    • Named entity recognition (NER)
• Customising the spaCy pipeline
  • Modifying spaCy pipelines
  • Processing texts efficiently
  • Adding custom attributes to spaCy objects
  • Writing processed texts to disk
  • Simplifying output for noun phrases and named entities
    • Merging noun phrases
    • Merging named entities
• Evaluating language models
  • What is a gold standard?
  • Measuring reliability manually
    • Step 1: Annotate data
    • Step 2: Calculate percentage agreement
    • Step 3: Calculate probabilities for each category
    • Step 4: Estimate chance agreement
  • Cohen’s kappa as a measure of agreement
  • Evaluating the performance of language models
• Managing textual data using pandas
  • Importing data to pandas
    • Importing data from a single file
    • Importing data from multiple files
  • Examining DataFrames
  • Extending DataFrames
  • Saving DataFrames