Stat 133: Concepts in Computing with Data

This course is NOT:

• A course in Computational Statistics,

  For example, we will not cover various algorithms for computing least squares solutions and inverting matrices

• A course in Applied Statistics

  For example, we will not learn a battery of statistical methods such as ordinary least squares, weighted least squares, and general linear models, etc.

• A course in Mathematical Statistics

  For example we will not cover the expectation, variance, and large sample properties of least squares estimators.

This course will address:

Various aspects of computing for statistics,
NOT the computational aspects of statistics.

DATA Technologies - You will need to work to get data, work with data, and work with the data ``owners"

Learn how to think about the data process

• data acquisition - Input/output, regular expressions
• data cleaning and verification - exploratory data analysis
• data organization - data frames, XML, databases
• data as a pseudo population - bootstrap, cross-validation,

An Example: SPAM

• SPAM = Unsolicited, mass, junk email.

• 1#1 of electronic mail is SPAM

• Offensive, time-consuming

• Want to be able to identify SPAM before we read it.

Return-Path: whisper@oz.net
Delivery-Date: Fri Sep  6 20:53:36 2002
From: whisper@oz.net (David LeBlanc)
Date: Fri, 6 Sep 2002 12:53:36 -0700
Subject: [Spambayes] Deployment
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEJABCAB.tim.one@comcast.net>
Message-ID: <GCEDKONBLEFPPADDJCOECEHJENAA.whisper@oz.net>

You missed the part that said that spam is kept in the "eThunk" and was
viewable by a simple viewer for final disposition?

Of course, with Outbloat, you could fire up PythonWin and stuff the spam
into the Junk Email folder... but then you loose the ability to retrain on
the user classified ham/spam.

David LeBlanc
Seattle, WA USA

> -----Original Message-----
> From: spambayes-bounces+whisper=oz.net@python.org
> [mailto:spambayes-bounces+whisper=oz.net@python.org]On Behalf Of Tim
> Peters
> Sent: Friday, September 06, 2002 12:24
> To: spambayes@python.org
> Subject: RE: [Spambayes] Deployment
>
> [Guido]
> > ...
> > - A program that acts both as a pop client and a pop server.  You
> >   configure it by telling it about your real pop servers.  You then
> >   point your mail reader to the pop server at localhost.  When it
> >   receives a connection, it connects to the remote pop servers, reads
> >   your mail, and gives you only the non-spam.
>
> FYI, I'll never trust such a scheme:  I have no tolerance for false
> positives, and indeed do nothing to try to block spam on any of my email
> accounts now for that reason.  Deliver all suspected spam to a Spam folder
> instead and I'd love it.
> _______________________________________________
> Spambayes mailing list
> Spambayes@python.org
> http://mail.python.org/mailman-21/listinfo/spambayes

Statistical problem:

Goal is to use statistical methodology to filter our mail.

• Get sample, classified messages
• Convert to response, predictors data frame
• Fit statistical model to data
  • use information from mail headers (i.e. sender, routing information, date, return address, etc.)
  • use information in the content of the message body
• Tune Type I and Type II errors
  • Type I error: reject regular message as SPAM
  • Type II error: accept SPAM as regular message.
• Deploy classifier as filter.

Another Example: Web Logs

• Each request to a Web site for a page is a hit
• Each hit recorded by Web server in a Web log
• Provides information about site
  • popular pages
  • common navigation paths through sites
  • busy times
  • resource usage and allocation
  • pre-fetching of pages
  • efficient proxy information

DATA

nchass03.telenet-ops.be - - [28/Dec/2003:06:33:55 -0600] "GET / HTTP/1.1" 200 718
nchass03.telenet-ops.be - - [28/Dec/2003:06:34:03 -0600] "GET /R.css HTTP/1.1" 200 658
nchass03.telenet-ops.be - - [28/Dec/2003:06:34:13 -0600] "GET /logo.html HTTP/1.1" 200 244
nchass03.telenet-ops.be - - [28/Dec/2003:06:34:23 -0600] "GET /navbar.html HTTP/1.1" 200 1418
nchass03.telenet-ops.be - - [28/Dec/2003:06:34:23 -0600] "GET /banner.shtml HTTP/1.1" 200 3185
nchass03.telenet-ops.be - - [28/Dec/2003:06:34:33 -0600] "GET /Rlogo.jpg HTTP/1.1" 200 8793
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:06 -0600] "GET /contrib/extra/dcom HTTP/1.1" 301 342
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:06 -0600] "GET /contrib/extra/dcom/ HTTP/1.1" 200 1404
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:06 -0600] "GET /icons/blank.gif HTTP/1.1" 200 148
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:06 -0600] "GET /contrib/extra/dcom/ HTTP/1.1" 200 1404
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:06 -0600] "GET /icons/blank.gif HTTP/1.1" 200 148
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:06 -0600] "GET /icons/binary.gif HTTP/1.1" 200 246
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:06 -0600] "GET /icons/back.gif HTTP/1.1" 200 216
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:07 -0600] "GET /icons/compressed.gif HTTP/1.1" 200 1038
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:07 -0600] "GET /icons/text.gif HTTP/1.1" 200 229
pcp07845748pcs.wilmsc01.tn.comcast.net - - [28/Dec/2003:06:36:15 -0600] "GET /contrib/extra/dcom/ReadMe.txt HTTP/1.1" 200 12247
nchass03.telenet-ops.be - - [28/Dec/2003:06:37:15 -0600] "GET /bin/windows HTTP/1.1" 301 335
nchass03.telenet-ops.be - - [28/Dec/2003:06:37:32 -0600] "GET /bin/windows/ HTTP/1.1" 200 1353
pr3-ts.telepac.pt - - [28/Dec/2003:06:38:53 -0600] "GET /bin/linux/debian/dists/sarge/main/binary-i386/Packages.gz HTTP/1.1" 200 2994
pr3-ts.telepac.pt - - [28/Dec/2003:06:39:02 -0600] "GET /bin/linux/debian/dists/sarge/main/binary-i386/Packages.gz HTTP/1.1" 200 2994
pr3-ts.telepac.pt - - [28/Dec/2003:06:39:17 -0600] "GET /bin/linux/debian/dists/sarge/main/binary-i386/Packages.gz HTTP/1.1" 200 2994
nchass03.telenet-ops.be - - [28/Dec/2003:06:39:28 -0600] "GET /bin/windows/base HTTP/1.1" 301 340
nchass03.telenet-ops.be - - [28/Dec/2003:06:39:47 -0600] "GET /bin/windows/base/ HTTP/1.1" 200 2167
nchass03.telenet-ops.be - - [28/Dec/2003:06:41:28 -0600] "GET /bin/windows/base/README.rw1081 HTTP/1.1" 200 11036
ip142177159155.ns.aliant.net - - [28/Dec/2003:06:42:22 -0600] "GET /robots.txt HTTP/1.1" 404 294
p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:40 -0600] "GET / HTTP/1.1" 200 718p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:41 -0600] "GET /R.css HTTP/1.1" 200 658
p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:41 -0600] "GET /logo.html HTTP/1.1" 200 244
p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:41 -0600] "GET /navbar.html HTTP/1.1" 200 1418
p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:42 -0600] "GET /banner.shtml HTTP/1.1" 200 3185
p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:41 -0600] "GET /navbar.html HTTP/1.1" 200 1418
p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:42 -0600] "GET /banner.shtml HTTP/1.1" 200 3185
p38.nas2.is4.u-net.net - - [28/Dec/2003:06:43:43 -0600] "GET /Rlogo.jpg HTTP/1.1" 200 8793
lj1229.inktomisearch.com - - [28/Dec/2003:06:45:34 -0600] "GET /robots.txt HTTP/1.0" 404 283

Steps

• Collapse sequence of lines into top-level page request
• Find included pages and read HTML.
• Find links in pages to get connectivity.
• Analyze connectivity and hits.
• Explore data graphically - mining.
• Pre-fetching model: fit 2-step Markov models for transitions between pages.
• Evaluate fit.
• Deploy pre-fetching model: Client or Server?

Goals

• Familiarity with statistical computing tools for
  • data acquisition
  • data processing
  • data visualization
  • data simulation
  • data as a pseudopopulation: resampling

• How to think about approaching problems related to computing on data
• Learn about currently emerging technologies
• Be able to intelligently discuss different technologies and tools, knowing when to use them and what are the trade-offs
• Be able to find out about and learn other technologies

Goals of the Course

• Focus: use existing software and functionality for context-specific analyses.
• Learn about: box of tools and how to use them to create things, and even build new tools.
• De-emphasize: understanding the existing algorithms.
• Understanding fundamental algorithms is important if you need to
  • recreate them in a new language
  • use them in new ways when developing new algorithms.

• Practical: how statistical methodology is used in Industry, Laboratory, Research

• Focus: overall task not just on the application of specific statistical methodology

General Information

• Instructor
  • Deborah Nolan
  • Office: 395 Evans
  • nolan@stat.berkeley.edu
  • Office Hours:
• GSI
  • Gang Liang
  • Office: 397 Evans
  • liang@stat.berkeley.edu
  • Office Hours:

• Lab meets on Fridays: 2-3 or 3-4 in 330 Evans

Course Materials

• There is NO textbook for the class.
  No single book covers it all!

• I'll prepare "chapters" going into more details than in class.

• Distributed via the Web at
  www.stat.berkeley.edu/~nolan/stat133/

• Many links to resources on the Web as you need them.

• We will use R as the primary computational environment

• R manuals
  • An Introduction to R cran.r-project.org/doc/manuals/R-intro.pdf
  • R Data Import/Export cran.r-project.org/doc/manuals/R-data.pdf
  • R Language Definition cran.r-project.org/doc/manuals/R-lang.pdf

  • On-line user guides to R, on-line help

  • S Programming, Venables & Ripley
  • Spector's course www.stat.berkeley.edu/classes/s296/

Computing Resources

• Statistical Computing Facilities (SCF) - networked computers running Unix
• Undergraduate computer laboratory
  • 342 Evans and 432 Evans
  • Open 8am to 6pm Monday through Friday.
  • Remote access through ssh
• Account from GSI on Friday
• Mailing lists: archive, post.
  • Class mailing list

Grading

Participation	10%	Class mailing list and in class
Homeworks	30%	5 Short computing assignments
Projects	60%	4 Parts each worth 15%
		May be done in groups of 2
		SPAM, Web site access, Network intrusion detection
		E-mail connectivity and social networks

Expectations: Although there is no computing, probability, or statistics prerequisite for this course, there is an expectation that you have the

• Curiosity,
• Initiative
• and Motivation

to Explore on your own and Learn as needed.