1. Access and Preprocess Text¶
The very first step of NLP processing chains is to access text. Some sources already contain text in a clean form, others, e.g. Websites, not only contain raw-text but also markup, images, tables, etc. In the latter case the challenge of preprocessing is to extract the raw text from the not-relevant parts. Moreover, preprocessing contains also the task of segmentation, i.e. the transformation of a possibly long text string into a list of sentences or words.
This chapter demonstrates how raw text can be accessed from:
local text-files
online text-files
online API’s, such as e.g. Twitter
Moreover, the process of crawling raw-text from
html files
RSS feeds is shown.
Corresponding preprocessing methods, e.g. for segmentation of strings into lists of words and lists of sentences are also demonstrated.