App store purchases

How can you get information about what apps you have and what you've spent on them?

• Look at the files in the "Mobile Applications" folder.

• Print from the Apps view in iTunes (use the Song listing option to get the most information).

• Use Spentonapps (I'm sure you're very enthusiastic about giving them your email and password so they can comb through your email account.)

• Look at your purchase history via iTunes 20 purchases at a time.

• Do some tedious email-mining.

Naturally, I choose the most tedious method.

I searched my gmail for iTunes receipt messages and applied a label to them. I don't usually store my email locally, so I set up gmail and apple's mail client to IMAP sync only one of my gmail labels (it would take way too long to download all messages). I figured out how to do this thanks to a comment by Matt at StackExchange:

if you have a lot of email, turn on the "Advanced IMAP Controls" Labs feature, then, under "Manage Labels", turn off IMAP access for all the folders except the one you want to download.

Thanks Matt! Once I had searched for the iTunes receipt messages, I applied a label to them and downloaded only those messages.

Next, I selected all the messages and saved them as plain text in a file called "receiptPlainText.txt". Then, some nasty bash:

cat receiptPlainText.txt | grep -v "Order Total:" | grep -B1 -A1 "Report a Problem" >purchasesOnly.txt

b=`cat purchasesOnly.txt | grep -v "Report a Problem"| sed 's/^Free/\$0.00/' | sed 's/, v[^,]*//' | tr -d ","`

echo $b | tr -d "*" | sed 's/ -- /\*/g' | tr "*" "\n" | grep "Seller"| sed 's/ Seller:/HBPH/' | sed 's/ \$\([0-9\.]\{4\}\)/, \1/g' | sed 's/HBPH/,/g' | sed 's/^/\"/' | sed 's/$/\"/' | sed 's/, /\", \"/g' >appPurchases.csv

The breakdown:

grep -v "Order Total:"

• remove prices that are totals

grep -B1 -A1 "Report a Problem"

• identify regions of text that are individual purchases

b=`

• remove newlines by saving the text as a variable

grep -v "Report a Problem"|

• remove unneeded line

sed 's/^Free/\$0.00/'

• make line structures comparable

sed 's/, v[^,]*//'

• remove version info

tr -d ","`

• remove commas

tr -d "*"

• remove any asterisks in app names

sed 's/ -- /\*/g'

• put a marker between records

tr "*" "\n"

• replace marker with newline

grep "Seller"

• remove records that aren't apps (all apps have sellers)

sed 's/ Seller:/HBPH/'

• replace Seller with marker for comma insertion

sed 's/ \$\([0-9\.]\{4\}\)/, \1/g'

• insert comma before price; remove $

sed 's/HBPH/,/g'

• replace marker with comma

sed 's/^/\"/' | sed 's/$/\"/' | sed 's/, /\", \"/g'

• put quotes around each field

As a side note, people get huffy about using cat before a command that can simply take a file name as an argument (the "Useless Use of Cat" award). However, I find it useful when I'm working on commands in a modular way. If I need to move the first grep statement around, I don't need to move the file name too.

I analyzed my data, and wow, I've bought a lot of apps. I couldn't bear to leave the left axis on, sorry.

It would be nice to have the date purchased too.

Law school:predatory lending ?=science training:pyramid scheme?

The New York Times profiles some unemployed, debt-encumbered lawyers and discusses how law schools report the status of their graduates in a misleading way. These reports promote the perception that going to law school is a good investment.

Catchy quote:

“Enron-type accounting standards have become the norm...Every time I look at this data, I feel dirty.” - law school professor William Henderson of Indiana University

The article reminds me of several recent articles pointing out the crummy conditions for people with science PhDs and how there too the interests of the educating institutions are at odds with those of the applicants and trainees. In the case of science, this is the result of how our academic science research and training is organized: a structure not infrequently called a pyramid scheme.

A faculty person who leads a lab is high on the pyramid, and the majority of the work is done by trainees (postdocs and graduate students) who are lower on the pyramid and mostly want to eventually lead their own lab. This structure was better suited to the expansive early days of civilian science (mid-1900s to 1970ish), because the the whole pyramid was growing, and trainees were relatively more likely to be able to move up the pyramid. When academic research expansion slowed, there were fewer new jobs created. For existing labs to continue, they needed a steady supply of trainees, but there was less demand for those people after they were trained.

What is the outcome? Between 1973 and 1996, the percent of scientists NOT working in permanent, full-time positions rose from 7% to 21%. This measurement was made among scientists who earned their PhDs 9-10 years previously, the assumption being they should no longer be in short-term training positions like a postdoctoral fellow positions.

(This statistic is based on figure 3.12 "Fraction of US life-science PhDs not holding permanent full-time jobs in science or engineering, 1973, 1985, and 1995" in Trends in the Early Careers of Life Scientists. National Research Council (US) Committee on Dimensions, Causes, and Implications of Recent Trends in the Careers of Life Scientists.

Washington (DC): National Academies Press (US); 1998.)

While this seems like an unsustainable system, at least PhD scientists usually don't accumulate hundreds of thousands of dollars in graduate school debt . Faint praise indeed.

Good girl dinette

I ate at Good girl dinette tonight with KC and MM. Yummy Vietnamese food, simple and delicious. House-made sodas, very good spring rolls, good mushroom sandwich. Crummy but friendly service.

The market for used iPhones

This beanplot shows the distribution of prices for which used 32GB 3GS iPhones were sold on eBay. A beanplot shows "a density trace, which is mirrored to form a polygon shape." Overlaid is "a one-dimensional scatter plot show[ing] all the individual measurements." When two phones were sold for the same price, the line is twice as wide, and so on.

Here's the R code that produced the plot

price_and_shipping=read.table("iphoneListingsPricesOnly.txt")
allprices= price_and_shipping
$V1+ price_and_shipping$V2
allprices=sort(allprices)
outlierCount=round(length(allprices)*.05,0)
prices=allprices[(outlierCount+1):(length(allprices)-outlierCount)]
library(beanplot)
png(filename = "priceDistribution.png")
beanplot(prices,method="stack",axes=TRUE,what=c(0,1,0,1),log="",main="Beanplot distribution of prices")
dev.off()

code formatting kludged together from output from Pretty R at inside-R.org -- i had to hack at it to get it to work on blogspot, and that made it uglier than it should be. normally they make (as per the name) pretty stuff.

Assessing prices on eBay

I recently listed my old iPhone for sale on eBay. I wanted to know how much phones were going for. But I didn't just want to peruse the prices; I wanted to plot them and analyze them. Here are the steps I took. They are ugly, but they worked on my system, which is Mac OS X Leopard, Firefox 3, and the Table2Clipboard extension (which you don't really need). I'm interested to hear about better ways to do this.

Do an eBay search.

• search text: iphone 3gs 32
• Preferences: Completed listings
• Preferences: Used
• show 200 results

Copy results to a local file

• create dedicated directory like /Users/exampleUser/ebaySoldPrice
• copy results tables only:

• select a word in the first listing
• from the right click menu, choose table2clipboard/select table
• scroll to end of page,
• shift click in last item to extend selection to include all items,
• right click and choose View selection source
• it takes a while for this window to populate
• Copy, paste and save text as "ebay_search_results.txt"

Clean up the results

• Open a terminal window and type the following commands (you may have to fix line breaks if you copy this text)



• cd /Users/exampleUser/ebaySoldPrice/





• cat ebay_search_results.txt  | sed 's/~//g' | 
  sed 's/<table class="li" r="/~/g' |
   tr  "~" "n" | sed 's/ shipping.*$//' |
    sed 's/([0-9]*).*class="vip">/1~/' |
    grep Sold   | sed 's/<[^$]*$/~/'  | 
  sed 's/Free shipping/$0/' |  sed 's/<[^$]*$/~/' |
   sed 's/<.*$/~na/' | tr "~" "t" 
  >ebay_search_results_cleaned.txt

here's how that breaks down:

• sed s'/\~//g'

                  ### remove any tildes in ad title
  

• tr  "~" "\n"

                  ### replace tildes with newline
  

• sed 's/<table class="li" r="/~/g'

                  ### replace table starts with tildes
  

• sed 's/ shipping.*$//'

                  ### get rid of text after shipping
  

• sed 's/\([0-9]*\).*class\=\"vip\"\>/\1\~/' |

                  ### keep the initial row number, deletes everything else until the title
  

• grep Sold

                  ### limit to items that sold
  

• sed 's/\<[^$]*\$/~/'  |

                  ### delete text between title and price
  

• sed 's/Free shipping/\$0/'

                  ### give free a numerical value
  

• sed 's/\<[^$]*\$/~/'  |

                  ### delete text between price and shipping price
  

• sed 's/<.*$/~na/'

                  ### if price doesn't have a dollar amount, delete text and give it an NA value
  

• tr "~" "\t"

                  ### replace tildes with tabs
  

• >ebay_search_results_cleaned.txt

                  ### direct the output into a file
  

Viet Noodle Restaurant in Atwater Village

photo by seanbonner

I love the spare aesthetic of this restaurant. I ate here today. The service was abysmally slow, but friendly. I had the vegetable soup, which was delicious. MM had the Beef Pho, which he said was fine, but was annoyed that he hadn't remembered it was nothing special.

Rose bowl post-parade float viewing

photos 1,3 and 4 by modmodworld

The day after the parade, I went to see the floats. It hadn't been on my LA bucket list, but it was urged, so I went. The float I liked most reminded me of the pottery design Star Glow (left), and thematically of two other patterns I like a lot: Salem Free Form Hop Scotch Pink and Jack Straw's Constellation (3rd and 4th from left). The photos are by Flickr user modmodworld who has a tremendous collection of photos of mid century dinnerware.