Regular Expression

These rely on being at the start of the string
What about finding a string anywhere in a line.
What about matching a string pattern anywhere in a line.
- an A followed by a C OR a G followed by a T and two or more of these patterns in succession
- occurrences of the same word twice
  - a a
  - the the
Matching a word, not just a string. cat in "cute cat", "mail catalogs", "bad cat!".
What about extracting part of a string somewhere in a line.
extract the value of the boundary string in the line

Content-Type: multipart/mixed; boundary="----ack1234"
extract the date in

for <jm@localhost>; Thu, 25 Jul 2002 06:05:36 -0400 (EDT)

What's this to do with Statistics

Email log information

Feb  3 23:41:18 jessie sendmail[16406]: [ID 801593 mail.info] i144fID8016406: from=<EEQXMCKJDFP@charter.com>, size=13570, class=0, nrcpts=1, msgid=<JXHYECDLQWVAEUZKFHGGTET@verizon.net>, proto=ESMTP, daemon=MTA, relay=guard.research.bell-labs.com [135.104.2.10]
Feb  3 23:41:19 jessie imapd[16410]: [ID 806292 mail.debug] imap service init from 135.104.12.62
Feb  3 23:41:19 jessie imapd[16410]: [ID 234311 mail.info] Login user=kc host=kc-lap.research.bell-labs.com [135.104.12.62]
Feb  3 23:41:19 jessie sendmail[16407]: [ID 801593 mail.info] i144fID8016406: to=<iis@jessie.research.bell-labs.com>, delay=00:00:01, xdelay=00:00:00, mailer=local, pri=43820, dsn=2.0.0, stat=Sent
Feb  3 23:41:49 jessie imapd[16410]: [ID 533678 mail.info] Logout user=kc host=kc-lap.research.bell-labs.com [135.104.12.62]
Feb  3 23:42:19 jessie imapd[16476]: [ID 806292 mail.debug] imap service init from 135.104.12.62
Feb  3 23:42:19 jessie imapd[16476]: [ID 234311 mail.info] Login user=kc host=kc-lap.research.bell-labs.com [135.104.12.62]
Feb  3 23:42:29 jessie sendmail[16478]: [ID 801593 mail.info] i144gTD8016478: from=<fE@henly.f9.co.uk>, size=7141, class=0, nrcpts=1, msgid=<b17f01c3ead8$1ae6f34b$4f6f7a11@S08aYfV>, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=guard.research.bell-labs.com [135.104.2.10]
Feb  3 23:42:29 jessie sendmail[16479]: [ID 801593 mail.info] i144gTD8016478: to=<mitra@jessie.research.bell-labs.com>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=37394, dsn=2.0.0, stat=Sent
Feb  3 23:42:37 jessie sendmail[16488]: [ID 801593 mail.info] i144gbD8016488: from=<owner-lds-gems@LISTS.LDS.ORG>, size=2896, class=0, nrcpts=1, msgid=<5.1.1.6.2.20040203062440.02c045e0@mail.mstar.net>, proto=ESMTP, daemon=MTA, relay=guard.research.bell-labs.com [135.104.2.9]
Feb  3 23:42:37 jessie sendmail[16489]: [ID 801593 mail.info] i144gbD8016488: to=<nuzman@jessie.research.bell-labs.com>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=33139, dsn=2.0.0, stat=Sent
Different fields in different lines.
Data are in the different fields that are context dependent.
Exclude lines that are system information, comments, etc.

Web log information

169.237.46.168 - - [26/Jan/2004:10:47:58 -0800] "GET /stat141/Winter04 HTTP/1.1" 301 328 "http://anson.ucdavis.edu/courses/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
169.237.46.168 - - [26/Jan/2004:10:47:58 -0800] "GET /stat141/Winter04/ HTTP/1.1" 200 2585 "http://anson.ucdavis.edu/courses/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
169.237.46.168 - - [26/Jan/2004:10:47:58 -0800] "GET /stat141/Winter04/Class.css HTTP/1.1" 404 1227 "http://winnie.ucdavis.edu/stat141/Winter04/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
169.237.46.168 - - [26/Jan/2004:10:47:58 -0800] "GET /stat141/Winter04/ucd1.gif HTTP/1.1" 200 6122 "http://winnie.ucdavis.edu/stat141/Winter04/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
169.237.46.168 - - [26/Jan/2004:10:48:06 -0800] "GET /stat141/Winter04/annnouncements.htm HTTP/1.1" 404 1227 "http://winnie.ucdavis.edu/stat141/Winter04/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
169.237.46.168 - - [26/Jan/2004:10:48:07 -0800] "GET /stat141/Winter04/Class.css HTTP/1.1" 404 1227 "http://winnie.ucdavis.edu/stat141/Winter04/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
169.237.46.173 - - [26/Jan/2004:11:03:07 -0800] "GET /stat141/Winter04/discussion2.html HTTP/1.1" 404 1056 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
169.237.46.173 - - [26/Jan/2004:11:03:12 -0800] "GET /stat141/Winter04/ HTTP/1.1" 200 2585 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
169.237.46.173 - - [26/Jan/2004:11:03:12 -0800] "GET /stat141/Winter04/Class.css HTTP/1.1" 404 1227 "http://winnie.ucdavis.edu/stat141/Winter04/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
169.237.46.173 - - [26/Jan/2004:11:03:12 -0800] "GET /stat141/Winter04/ucd1.gif HTTP/1.1" 200 6122 "http://winnie.ucdavis.edu/stat141/Winter04/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
169.237.46.173 - - [26/Jan/2004:11:03:15 -0800] "GET /stat141/Winter04/handouts.html HTTP/1.1" 200 558 "http://winnie.ucdavis.edu/stat141/Winter04/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
169.237.46.173 - - [26/Jan/2004:11:03:17 -0800] "GET /stat141/Winter04/Handouts/discussion2.html HTTP/1.1" 200 6100 "http://winnie.ucdavis.edu/stat141/Winter04/handouts.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
169.237.46.173 - - [26/Jan/2004:11:03:20 -0800] "GET /stat141/Winter04/Handouts/discussion3.html HTTP/1.1" 200 3003 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
169.237.99.188 - - [26/Jan/2004:11:03:47 -0800] "GET /stat141/Winter04 HTTP/1.1" 301 328 "http://www.stat.ucdavis.edu/courses/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030120"
- Machine of requester
- Date
- Command, file, protocol (HTTP or FTP)
- return status
- number of bytes returned
- referring page
- browser information.

Regular Expressions

Language for expressing string patterns.
Simplest pattern: literal string
"From" in "From bob@us.gov"
Matches "From" in any part of the text <b>From our partners:</b>
Match "cat" in "dogs and cats", "My cat", "a catalog".
Need way to express patterns:
- white space,
- word boundaries
- sets of characters
- beginning or end of line
- alternatives - AC or GT
- number of matches.
modify text by substituting values for patterns.
```
    gsub("^From", "X-From:", h[1])
```
Matching is often, by default, line oriented.

R Functions for Regular Expressions

grep()
regexpr()
gsub()
sub()

Similar facilities available in many other programming environments. Perl, Java, Matlab, C libraries, ...
Different dialects and extensions Share a common core.

Meta-Characters

Patterns are expressed using meta-characters not interpreted as literal characters.
^ is a meta-character meaning "start of the line"
$ is meta-character meaning "end of the line"
Find empty lines:
```
grep("^$", x)
```
Match lines in WebLogs that end with -

proxy-6.daimler-benz.com - - [29/Dec/2003:06:35:48 -0600] "GET /R.css HTTP/1.1" 304 -
```
grep("-$", x)
```
Match lines in WebLogs that returned 0 bytes.

220.73.165.205 - - [29/Dec/2003:06:33:01 -0600] "HEAD /src/contrib/xtable_1.1-2.tar.gz HTTP/1.0" 404 0
```
grep(" 0$", x)
```
What can go wrong with this? i.e. what can we miss.

Escaping Meta-Characters

What if we want to match a ^ in a string? or a $?
Need to avoid interpretation of ^ or $ as a meta-character
Need to "escape" its meaning

 >  regexpr("\\^", "This has a caret ^.")

In other languages, just need a single \ $x =~ /\^/;

Alternatives

We often want to say "match this or this or this" To or From or Cc followed by a :
The | meta-character allows us to do this
To|From|Cc
Need to add the : to each of them To:|From:|Cc:
But don't want to repeat : each time Gets complicated with more complex expressions than ":"
Need to group (sub-) expressions Use the ( ) qualifiers around the sub-expression.
(To|From|Cc):
This will match "To:", "From:" and "Cc:".
How do we restrict match to beginning of the line?

Character Sets

In addition to alternative expressions, we often want a simpler form match any of the characters in the set {a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z, A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z}
Can write this is as (a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|A|B|C|D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z)
Cleaner form is [abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ]
The [ ] pair are meta-characters meaning a character set
The elements within the [ ] are the characters to match.
Now we can make our matching of WebLog lines that returned 0 bytes more precise.

220.73.165.205 - - [29/Dec/2003:06:33:01 -0600] "HEAD /src/contrib/xtable_1.1-2.tar.gz HTTP/1.0" 404 0
```
grep("[ \t]0$", x)
```

More on Character Sets

Expressing the alphabet, or all numbers is tedious to enumerate completely.
Can use ranges in many common cases [a-f] [A-Z] [0-9]
What if we want a - in the character set?
Built-in character sets for:
- All alphabetic and numeric (alnum)
- All alphabetic (alpha)
- Blank characters, i.e. space or tab (blank))
- Control characters (cntrl)
- Digits (digit)
- Printable character except space (graph)
- Lower case alphabetic characters (lower)
- Upper case alphabetic characters (upper)
- Printable characters (print)
- Punctuation characters (punct)
- White space (space)
- Hexadecimal digit (xdigit)
Used as [[:class-name:]]
regexpr("[[:digit:]]", "Some numbers and text: 1312 and bob")
Need to express "every character except the ones in this set"
Do this using the ^ meta-character in a new way (not start of line)
If first character inside the [ ] is a ^, we mean the complement of the set.
regexpr("[^0-9]", "10000e10")

Other Meta-Characters and Constructs

The single character . matches any character.
We also need to be able to specify how the number of times an expression need match
Expression by itself - exact match once.
```
 regexpr("a", "this is a sentence")
```
Expression followed by * - 0 or more matches.
```
 regexpr("a*", "this is aaaa sentence")
```
Expression followed by ? - 0 or 1 matches.
```
 regexpr("a?", "this is a sentence")
```
Expression followed by + - 1 or more matches.
```
 regexpr("f+", "aardvark")
```
What does ".*" match? What does "^.*$" match?

Additional Number of Matches

What about exactly 3 "a"s?
aaa
What about ACGT repeated 12 times? Very cumbersome to enumerate
(ACGT){12}
What about one of ACGT or ACCT 12 times? or 12 repetitions of either ACGT or ACCT?
And what of 12 or more times? (ACGT){12}(ACGT)*
Convenient expression: (ACGT){12,}
And for at least m but not more than n (where m <= n) pattern{m,n}
Combine with more complex sub-expressions
```
grep("(ACT|GG){2,5}", genes)
```
- pattern{m} - exactly m matches
- pattern{m,} - at least m matches
- pattern{m,n} - at least m, no more than n matches

Grouping and References

Use to create sub-expressions
Also can refer to these later in the regular expression use the group number 1, 2, ..., 9 Need to escape

regexpr("([a-z]+) \\1", c("Ha a sentence", "with the the?", "notnot here", "with the they","no match"))

regexpr("(\\<[a-z]+\\>) \\<\\1\\>", c("Ha a sentence", "with the the?", "notnot here", "with the they", "no match"))

Substitutions and Replacements

Instead of just matching, we want to replace patterns with other values.
Use gsub()
- Regular expression defining what to match
- expression of what to substitute for each match
- the vector of strings to process.
Example: gsub("^From[ \t]", "X-From:", x)
Replaces only the matching element. To remove other pieces in result, must match those expressions also.
```
gsub("from", "to", "Note from bob")
```

Web Log Substitution Example

Example: transform Web log line into comma-separated list of IP address, date, file, status, bytes Then access values with strsplit().
Transform

193.188.97.151 - - [29/Dec/2003:06:36:18 -0600] "GET /logo.html HTTP/1.1" 200 244

to

193.188.97.151, 29/Dec/2003:06:36:18, /logo.html, 200, 244

and then strsplit("193.188.97.151, 29/Dec/2003:06:36:18, /logo.html, 200, 244", ",")
(.*) - - \\[(.*) [-+][0-9]+\\] "(GET|POST) (.*) (HTTP|FTP)(/1.[01])?" ([0-9]+) (-|[0-9]+)
Substitution string:

\\1,\\2,\\4,\\7,\\8

Skip over the groups we need for the match, but not for the substitution.
Syntax available for not numbering a group, if necessary
Note that we have to make certain that the separator we use is not in any of the elements we want! We would like to get back the elements we match directly. Perl can do this.

Pitfalls of Regular Expressions

Regular expressions are greedy they match as much as they can!

Discard HTML tags?

gsub("<.*>", "", 'URL: <a  href="http://www.bob.org"><i>Bob</i></a><p>')
[1] "URL: "

Need to be more specific:

gsub("<[^<]*>", "", 'URL: <a  href="http://www.bob.org"><i>Bob</i></a><p>')
[1] "URL: Bob"

And what about a real < that is not part of the HTML, e.g. \\<?
They require experimentation and experience. Practice.