---
title: "Quant 3: Supplemental Materials for Week 1"
date: "16 February 2018"
output: html_document
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "##"
)
```

# Introduction to RMarkdown

Before we dive into our first coding session, let's become a bit more familiar with the programming tools used in this course.

We will write our annotated R code using [Markdown](http://rmarkdown.rstudio.com).

__Markdown__ is a simple formatting syntax to generate HTML or PDF documents. In combination with R, it will generate a document that includes the comments, the R code, and the output of running such code.

You can embed R code in chunks like this one:

```{r}
1 + 1
```

You can run each chunk of code one by one, by highlighting the code and clicking `Run` (or pressing `Ctrl + Enter` in Windows or `command + enter` in OS X). You can see the output of the code in the console right below, inside the RStudio window.

Alternatively, you can generate (or __knit__) an html document with all the code, comment, and output in the entire `.Rmd` file by clicking on `Knit HTML`.

You can also embed plots and graphics, for example:

```{r}
x <- c(1, 3, 4, 5)
y <- c(2, 6, 8, 10)
plot(x, y)
```

If you run the chunk of code, the plot will be generated on the panel on the bottom right corner. If instead you knit the entire file, the plot will appear after you view the html document.

Using R + Markdown has several advantages: it leaves an "audit trail" of your work, including documentation explaining the steps you made. This is helpful to not only keep your own progress organized, but also make your work reproducible and more transparent. You can easily correct errors (just fix them and run the script again), and after you have finished, you can generate a PDF or HTML version of your work.

We will be exploring R through R Markdown over the next few modules. For more details and documentation see <http://rmarkdown.rstudio.com>.

# Make sure R and RStudio are installed

Follow the instructions in the class material and install R and RStudio. If you feel more comfortable using the basic R terminal, skip the step of installing RStudio and the corresponding chunk.

Now run the following code to make sure that you have the current version of R.

```{r}
version$version.string
```

This chunk should return `R version 3.4.3 (2017-11-30)`.

```{r, eval=FALSE}
rstudioapi::versionInfo()$version
```

This chunk should print `1.1.383`.

```{r}
installed.packages()["quanteda", "Version"]
```

This chunk should print `1.0.0` (published at CRAN on 15/1/2018).

If any of those chunks do not print the correct version numbers, head back to the slides of the first lab session and follow the steps outlined there to install

# Basic string manipulation functions in R

Start by loading `quanteda`.

```{r message=FALSE}
library("quanteda")
```

There are several useful string manipulation functions in the R base library. In addition, there is the `stringr` package which provides an additional interface for simple text manipulation. The package will not be covered in this introduction but it is highly recommended if your work requires more than basic string manipulation.

The fundamental type (or `mode`) in which R stores text is the character vector. The most simple case is a character vector of length one. The `nchar` function returns the number of characters in a character vector.

```{r}
s1 <- 'my example text'
length(s1)
nchar(s1)
```

## Counting characters

The `nchar` function is vectorized, meaning that when called on a vector it returns a value for each element of the vector.

```{r}
s2 <- c('This is', 'my example text.', 'So imaginative.')
length(s2)
nchar(s2)
sum(nchar(s2))
```

We can use this to answer some simple questions about election manifestos by UK parties on immigration.

Which are the longest and shortest statements? We can query this using two functions, `nchar()` and `which.max()` and `which.min()`.

```{r}
which.max(nchar(data_char_ukimmig2010))
which.min(nchar(data_char_ukimmig2010))
```

## Extracting characters

Unlike in some other programming languages, it is not possible to index into a string in R:

```{r}
s1 <- 'This file contains many fascinating example sentences.'
s1[6:9]
```

To extract a substring, instead we use the `substr()` function. Using the help page from `?subst`, execute a call to `substr()` to return the characters from `s1` below from the 6th to the 9th characters.

```{r}
s1 <- 'This file contains many fascinating example sentences.'
substr(s1, 6, 9)
```

A note for you C, Python, Java, ... programmers: __R counts from 1, not 0__.

## Splitting texts and using lists

Often we would like to split character vectors to extract a term of interest. This is possible using the `strsplit` function. Consider the names of the inaugural texts corpus (`data_corpus_inaugural`):

```{r}
docnames(data_corpus_inaugural)
# returns a list of parts
parts <- strsplit(docnames(data_corpus_inaugural), '-')
years <- sapply(parts, function(x) x[1])
pres <- sapply(parts, function(x) x[2])
```

Examine the previous code carefully, as it uses list data types in R, which are something fundamentally important to understand. In **quanteda**, the `tokens` class of object -- created when you call `tokens()` on a character object or corpus -- is a type of list. Try it:

```{r}
toks <- tokens("This is a sentence containing some charactères français.")
```

Now examine the "structure" of that object -- assigned to `toks` -- using `str()`.

Try sending `toks` to the global environment, by simply typing its name in the console and pressing Enter. Can you explain why it looks the way that it does? Hint: You can examine all available "methods" for an object class using the `methods()` function. Try `methods(class = "tokens")`, and use the help function `?methods` to explain what you see.

## Joining character objects together

The `paste` function is used to join character vectors together. The way in which the elements are combined depends on the values of the `sep` and `collapse` arguments:

```{r}
paste('one','two','three')
paste('one','two','three', sep = '_')
paste(years, pres, sep = '-')
paste(years, pres, collapse = '-')
```

## Manipulating case

`tolower` and `toupper` change the case of character objects:

```{r}
tolower(s1)
toupper(s1)
```

These are also examples of "vectorized" functions: They work on vectors of objects, rather than just atomic objects. Try these functions on the character vectors below:

```{r}
s_vec <- c("Quanteda is the Best Text Package Ever, approved by NATO!",
           "Quanteda является лучший текст пакет тех, утвержденной НАТО!")
```

Try running `tolower()` on that vector. What results?

**quanteda** has its own, smarter lowercase function, called `char_tolower()`. Try it on `s_vec`. There is an option to preserve the acronym -- try it a second time while preserving the acronym `NATO` as uppercase. To find out how, read the fine manual (RTFM): `?char_tolower`.

Note how this works in English as well as in Russian thanks to the marvels of Unicode!

## Comparing character objects

Character vectors can be compared using the `==` and `%in%` operators:

```{r}
char_tolower(s1) == char_toupper(s1)
'apples' == 'oranges'
char_tolower(s1) == char_tolower(s1)
'pears' == 'pears'

c1 <- c('apples', 'oranges', 'pears')
'pears' %in% c1
c2 <- c('bananas', 'pears')
c2 %in% c1
```

## Preparing and pre-processing texts

It is common to "clean" texts before processing, usually by removing punctuation, digits, and converting to lower case.

"Cleaning" in **quanteda** takes through decisions made at the tokenization stage. In order to count word frequencies, we first need to split the text into words through a process known as **tokenization**. Look at the documentation for **quanteda**'s `tokens` command using the built in help function (`?` before any object/command). Use the `tokens` command on `data_char_sampletext` (a built-in data type in the **quanteda** package), and examine the results.

1.  Stemming.

    Stemming removes the suffixes using the Porter stemmer, found in the **SnowballC** library. The **quanteda** functions to invoke the stemmer are `char_wordstem()`, `tokens_wordstem()`, and `dfm_wordstem()`. Apply stemming to the `exampleString` and examine the results. Why does it not work, and what do you need to do to make it work? How would you apply this to the sentence-segmented vector?

2.  Applying pre-processing to the creation of a **dfm**.

    **quanteda**'s `dfm()` function by default applies certain "cleaning" steps to the text, which are not the defaults in `tokens()`. Create a dfm from `data_char_sampletext`. What are the differences between the steps applied by `dfm()` and the default settings for `tokens()`?

    ```{r}
    sample_corpus <- corpus(data_char_sampletext)
    (sample_dfm <- dfm(sample_corpus))
    ```

    Inspect the dimensions of the resulting objects, including the names of the words extracted as features. To get the "words" from a **quanteda** object, you can use the `featnames()` function:

    ```{r}
    featnames(sample_dfm) %>% head(20)
    ```

3.  Keywords in context.

    Use the `kwic` function to discover the context of the word "clean". Is this associated with environmental policy?

    Using the `data_corpus_irishbudget2010` object, examine the context for the word "Irish". What is its predominant usage?

    Examine the context of words *related to* "disaster". Hint: you can use the stem of the word along with setting the `regex` argument to `TRUE`. Execute a query using a pattern match that returns different variations of words based on "disaster" (such as disasters, disastrous, disastrously, etc.).

    Load the text of Herman Melville's *Moby Dick*. You can use the base R solution or alternatively install the **readtext** package and use its simpler interface. Use `kwic()` to search for "Ahab", and save this object. Send it to `textplot_xray()`.

    ```{r}
    # This is a base R solution to reading text from a URL
    mobydicktf <- paste(readLines("https://kenbenoit.net/assets/files/pg2701.txt"), collapse = "\n")

    # If you install the "readtext" package, you can use the following simpler code:
    # readtext::readtext("https://kenbenoit.net/assets/files/pg2701.txt")
    # Highly recommended if you plan to read 

    mobydickCorpus <- corpus(mobydicktf, docvars = data.frame(doc_id = "pg2701.txt"))

    # command to produce a kwic

    # command to produce the x-ray plot
    ```

4.  **Descriptive statistics**

    Compute descriptive statistics for the `data_corpus_irishbudget2010` object. Hint: `?summary`.


## A note about NAMESPACEs

In R's "S3" object-oriented system, functions of the same name can be written so as to dispatch different "methods" depending on the class of the object on which the function is called. This also explains why you get certain warning messages when you attach the **quanteda** package, e.g.

```{r}
detach("package:quanteda")
require(quanteda)
```

Here, the object `View` (a function) from the **utils** package -- which is one of the standard packages that is always attached when you start R -- has been superceded in priority in R's "namespace" by another object (also a function) called `View` from the **quanteda** package. Compare the two using `?View`, where you should see two versions listed.
