A word cloud where the x and y axes mean something

A word cloud with meaningful spatial properties.

Ok so I have now done two iterations on a better way to visualize term frequencies using R, ggplot2 and plyr. The first was ok but ugly, the second was better but still ugly.

How to read it:

• Frequency is segmented in to 20% quantiles
• The frequency is on the y axis
• Word size is proportional to frequency
• Words with similar frequency are in approximately alphabetical order from left to right.
• Color is still random (this could be better)

This one is now good enough that I will start using it in my own presentations and announce my retirement from the prestigious and highly-paid world of word-cloud improvement!

Here’s the code.

library(languageR)

# get english word freq data
data(english)
df <- english[,c("Word","WrittenFrequency")]
df <- df[sample.int(NROW(df),200),]
df <- unique(df)
df$freq <- df$WrittenFrequency/sum(df$WrittenFrequency)
qtiles <- quantile(df$freq, seq(0,1,.2))
twotiles <- quantile(df$WrittenFrequency, seq(0,1,10/NROW(df)))
qdf <- data.frame(cut = qtiles,quantile= as.numeric(strsplit(names(qtiles),"%")))
df$qtilerange <- cut(df$freq,breaks=qtiles,labels=F)
df$twotiles <- as.factor(cut(df$WrittenFrequency,breaks=twotiles,labels=F))
df$quantile <- qdf[(df$qtilerange+1),"quantile"]
df$quantilecut <- qdf[df$qtilerange,"cut"]
df <- df[order(df$quantile),]
df$quantile <- as.factor(df$quantile)
df$quantile <- reorder(df$quantile,NROW(df):1)
df$WordColor <- factor(sample.int(4,NROW(df),replace=T))
df <- df[!is.na(df$quantile),]

ticks <- ddply(df,c("quantile"),summarize,ticks=quantile(WrittenFrequency,c(.2,.8)))$ticks
ticks <- round(unique(c(max(df$WrittenFrequency),ticks)),2)

df <- ddply(df,c("twotiles"),summarize,
            Word=sort(Word),
            WordColor=WordColor, 
            WrittenFrequency=WrittenFrequency, 
            quantile=quantile,
            x=seq(-min(WrittenFrequency)/mean(WrittenFrequency),max(WrittenFrequency)/mean(WrittenFrequency),length.out=length(WrittenFrequency))
            )

library(ggplot2)
# frequency label on the yaxis # x axis is frequency scale  (log data in this example) # word name is shown in the facet label
p <- ggplot(df,aes(x=x,y=WrittenFrequency))
p <- p + geom_text(aes(label=Word,size=WrittenFrequency,color=WordColor),family="Courier",face="bold")
p <- p + opts(axis.text.x=theme_blank(), axis.title.x=theme_blank(),panel.grid.major=theme_blank()) 
p <- p + scale_y_continuous(breaks=ticks)
p <- p +  facet_grid(quantile~.,scales="free_y",space="free",labeller = label_both)
p + opts(strip.text.y = theme_text(angle = 0, size = 15, hjust = 0.5, vjust = 0.5),
         axis.text.y = theme_text(angle = 0, size = 15, hjust = 0.5, vjust = 0.5),
         axis.title.y = theme_blank(),
         legend.text=theme_blank(),legend.position = "none",
         title="Word Frequency")

Word cloud alternatives

This is an attempt to make word clouds more quantitative. It still needs more work in order to be an aesthetic competitor to the classic word cloud.

Here is an alternative to word clouds that makes it easier to get insights, but also has some of the aesthetic appeal of the traditional word cloud.
My first attempt at this looked pretty bad and this is not too much better, but hopefully someone else will help improve it.

library(languageR)

# get english word freq data
data(english)
df <- english[,c("Word","WrittenFrequency")]
df <- df[sample.int(NROW(df),500),]
df <- unique(df)
df$freq <- df$WrittenFrequency/sum(df$WrittenFrequency)
qtiles <- quantile(df$freq, seq(0,1,.2))
qdf <- data.frame(cut = qtiles,quantile= as.numeric(strsplit(names(qtiles),"%")))
df$qtilerange <- cut(df$freq,breaks=qtiles,labels=F)
df$quantile <- qdf[(df$qtilerange+1),"quantile"]
df$quantilecut <- qdf[df$qtilerange,"cut"]
df <- df[order(df$quantile),]
df$quantile <- as.factor(df$quantile)
df$quantile <- reorder(df$quantile,NROW(df):1)
df$WordColor <- factor(sample.int(5,NROW(df),replace=T))
df <- df[!is.na(df$quantile),]

ticks <- ddply(df,c("quantile"),summarize,ticks=quantile(WrittenFrequency,c(.2,.8)))$ticks
ticks <- round(unique(c(max(df$WrittenFrequency),ticks)),2)

rollfun <- function(x) {
  numb <- 10
  scale <- mean(x)/(max(x)-min(x)+0.01)
  tmp <- rnorm(1,0,scale)
  tmp <- ifelse( tmp < -numb, -numb,tmp)
  tmp <- ifelse( tmp > numb, numb,tmp)
  tmp
}

roll <- rollapply(df$WrittenFrequency,5,rollfun)
df$x <- c(roll,roll[1:(NROW(df)-NROW(roll))] )

library(ggplot2)
# frequency label on the yaxis # x axis is frequency scale  (log data in this example) # word name is shown in the facet label
p <- ggplot(df,aes(x=x,y=WrittenFrequency))
p <- p + geom_text(aes(label=Word,size=sqrt(WrittenFrequency),color=WordColor),family="Courier",face="bold")
p <- p + opts(axis.text.x=theme_blank(), axis.title.x=theme_blank(),panel.grid.major=theme_blank()) 
p <- p + scale_y_continuous(breaks=ticks)
p <- p +  facet_grid(quantile~.,scales="free",space="free",labeller = label_both)
p + opts(strip.text.y = theme_text(angle = 0, size = 15, hjust = 0.5, vjust = 0.5),
         axis.text.y = theme_text(angle = 0, size = 15, hjust = 0.5, vjust = 0.5),
         axis.title.y = theme_blank(),
         legend.text=theme_blank(),legend.position = "none",
         title="Word Frequency")

Stop squinting at word clouds in the hope of getting insights

Someone recently asked on twitter about about peoples' preferences for cloud generators in R.  
I replied that I thought the "null" word cloud generator was best. By this I mean that I think the word cloud is a bad visualization method. 
Why? Here is one article with a good perspective, but you can search for examples and see what insights you can get from word clouds; I think they usually obscure the insights. If you are trying to understand raw text then you really want to do better text mining rather than just word frequencies.  And if you want to just look at term frequencies, the word cloud is a very fuzzy way to go about it.

So the natural followup question is how to plot phrase/word frequency data.

Here is an example of the kind of thing that I usually do.  This is only for raw term frequency data (you will need to tabulate it yourself first, which is easy).  For real text mining analysis you can always use packages from the CRAN Task View. 

library(languageR)

# get english word freq data
data(english)
df <- english[,c("Word","WrittenFrequency")]
#reorder by freq for plotting
df <- df[order(-df$WrittenFrequency),]
df$Word <- reorder(df$Word,1:NROW(df))
#get the top 75 words
df <- head(df,75)

library(ggplot2)

# frequency label on the yaxis # x axis is frequency scale  (log data in this example) # word name is shown in the facet label
p <- ggplot(df,aes(x=WrittenFrequency,y=WrittenFrequency))
p <- p + geom_point(size=5)
p + facet_grid(Word~.,scales="free") +  opts(strip.text.y = theme_text(),axis.title.y= theme_blank())

There are lots of things you can do to make it fancier and prettier.  Does anyone have something better?

Stupid R tricks: using outer to create many data.frame subsets

Selecting subsets of a data.frame is easy in R if you define the predicates manually.
But if you need to define many conditions the standard slicing and subsetting methods
are cumbersome.

For this illustration I want to pick some large number of numerical ranges and label
all of the rows that match any of the predicates.

The key is using outer to match against many predicates and then checking that any of them was satisfied.

peaks <- pi*c(0,2,4,6,8,10)
low <- peaks - pi/4
high <- peaks + pi/4
ranges <- data.frame(low=low,high=high)

x<- seq(0,10*pi,0.01)
y<- cos(x)
df <- data.frame(x=x,y=y)

# given a vector x
# which elements are contained in one of the ranges
# defined by the high and low columns of the ranges data.frame
library(plyr)
inranges <- function(x, ranges)
{
  a<-outer(x,ranges$low, ">")
  b<-outer(x,ranges$high, "<")
  c<-a & b
  aaply(c,1,function(y) any(y) )
}

# I can now add a new column that indicates which rows matched
df$peaks <- inranges(df$x, ranges)

library(ggplot2)
p <- ggplot(df,aes(x=x,y=y))
p <- p + geom_point(aes(color=peaks))
p

#or I can subset the data to only the matching rows:

df.peaks <- subset(df,inranges(x,ranges))

p <- ggplot(df.peaks,aes(x=x,y=y))
p <- p + geom_point()
p

Data scientists: don’t let your company’s infrastructure make you soft

There has been a lot written about the skills needed to be a Data Scientist.  Not only should you be able to do these standard things:

1. Wrangle data (get, transform, persist)
2. Model (explore, explain and predict)
3. Take action (visualize, summarize, prototype)

…but I would argue that you should also be able to start with a bare machine (or cluster) and bootstrap a scalable infrastructure for analysis in short order. This does not mean you need to be able to administer a 1000-node hadoop cluster, but you should be able to set up a small cluster that can process TBs of log data into something that has business value.

For people who work for a big company it is easy to fall into the habit of using whatever infrastructure is available. Your IT department may have set up a hadoop cluster, there may be databases that are pre-configured and there are probably a lot of nice productivity tools that make it easier to analyze data at work.  It makes perfect sense for companies to provide these conveniences and it probably makes your job easier.  But it is also easy to get too cozy with this tool chain and come to rely on it.

In this series of posts I am going to talk about the analysis stack on my personal computers that help me do those things.

1. R (and RStudio)
2. MySQL
3. Hadoop (Scala, Cascading, scalding, scoobi)
4. …

It took me a while to get this set up but I have a goal of being able to start from scratch and install a complete working data science setup in 6 hours or less.

Simple enough to comprehend?

The Tea Party, Occupy Wallstreet and many other movements (that are not about human rights) share the same problems. In order to gain a big following they have to have very, very simple ideas at their core. Charles Stross wrote a nice blog post about a totally different topic but I am going to shamelessly quote out of context.

I think these ideas are mostly delusional because they rely on a fundamental misapprehension about the world around us — namely that we live in a society that can be made simple enough to comprehend.
Stross: insufficient data

Governing gets harder as the world gets more complex because our ability and desire to understand complexity is not growing exponentially.

Using transparency for data count intuition

This is an illustration of representing point count in a graphic using transparency. This is easy to do in ggplot2 if you use one of the barchart type of geoms.  However I think there are other situations where it would be useful to apply aesthetics based on point count.

Since Hadley did a lot of his canonical examples using this data I thought it would be helpful for comparing and contrasting.

This chart shows the distribution of the price/carat of diamonds segmented by quartile of carats and clarity.  The transparency shows how many diamonds each bar represents.  This makes it easy to see where the action is.

 library(ggplot2)
# create copy of diamonds
 df <- diamonds
# compute the quartiles of carat
 df$carat.qtiles <- cut(df$carat,unlist(quantile(df$carat)),include.lowest=T)
# plot the probability distribution of price/carat, faceted by clarity and carat quartile
 # key point: using the count per bar to set the alpha level. This lets you see how much
 # data is represented by each bar (it would be nice to be able to do this
 # anytime an aggregate is done...boxplots, bins, etc.)
 p <- ggplot(data=df, aes(x=price/carat,y=..count../sum(..count..)))
 p <- p + geom_histogram(aes(alpha=..count..),binwidth=1000) +facet_grid(clarity~carat.qtiles)
 p

Currently in ggplot2 this method will only work if the ..output.. variables related to count are available. There are a number of areas that could benefit from this capability.  It should also be easy to add more output variables to the elements of ggplot for which this behavior would be natural.

1. geom_boxplot:  Geoms that aggregates multiple points are good candidates for this
2. facet_*: It would be interesting to be able to add a visual cue to each facet to show how many points are in each.
   1. The most appealing idea on this so far is to enable scaling of the facet area by point count (or other things).
   2. Ordering of the facets by point count would also be extremely useful.
   3. Thresholding by count.  This would be great to easily chop low-signal facets and keep the visualization clean.
   4. Other half-baked ideas include background color, alpha box border…