Tokenize R Code

Tools for tokenizing R code.

Usage

tokenize_file(path)

tokenize_string(string)

tokenize(file = "", text = NULL)

Arguments

file, path

A file path.

text, string

R code as a character vector of length one.

Value

A data.frame with the following columns:

value	The token's contents, as a string.
row	The row where the token is located.
column	The column where the token is located.
type	The token type, as a string.

Note

Line numbers are determined by existence of the \n line feed character, under the assumption that code being tokenized will use either \n to indicate newlines (as on modern Unix systems), or \r\n as on Windows.

Examples

tokenize_string("x <- 1 + 2")
#>   value row column       type
#> 1     x   1      1     symbol
#> 2         1      2 whitespace
#> 3    <-   1      3   operator
#> 4         1      5 whitespace
#> 5     1   1      6     number
#> 6         1      7 whitespace
#> 7     +   1      8   operator
#> 8         1      9 whitespace
#> 9     2   1     10     number