Friday, January 28, 2011

App store purchases

How can you get information about what apps you have and what you've spent on them?

  • Look at the files in the "Mobile Applications" folder.
  • Print from the Apps view in iTunes (use the Song listing option to get the most information).
  • Use Spentonapps (I'm sure you're very enthusiastic about giving them your email and password so they can comb through your email account.)
  • Do some tedious email-mining.

Naturally, I choose the most tedious method.

I searched my gmail for iTunes receipt messages and applied a label to them. I don't usually store my email locally, so I set up gmail and apple's mail client to IMAP sync only one of my gmail labels (it would take way too long to download all messages). I figured out how to do this thanks to a comment by Matt at StackExchange:
if you have a lot of email, turn on the "Advanced IMAP Controls" Labs feature, then, under "Manage Labels", turn off IMAP access for all the folders except the one you want to download.
Thanks Matt! Once I had searched for the iTunes receipt messages, I applied a label to them and downloaded only those messages.

Next, I selected all the messages and saved them as plain text in a file called "receiptPlainText.txt". Then, some nasty bash:

cat receiptPlainText.txt | grep -v "Order Total:" | grep -B1 -A1 "Report a Problem" >purchasesOnly.txt

b=`cat purchasesOnly.txt | grep -v "Report a Problem"| sed 's/^Free/\$0.00/' | sed 's/, v[^,]*//' | tr -d ","`

echo $b | tr -d "*" | sed 's/ -- /\*/g' | tr "*" "\n" | grep "Seller"| sed 's/ Seller:/HBPH/' | sed 's/ \$\([0-9\.]\{4\}\)/, \1/g' | sed 's/HBPH/,/g' | sed 's/^/\"/' | sed 's/$/\"/' | sed 's/, /\", \"/g' >appPurchases.csv

The breakdown:

grep -v "Order Total:"
  • remove prices that are totals
grep -B1 -A1 "Report a Problem"
  • identify regions of text that are individual purchases

  • remove newlines by saving the text as a variable
grep -v "Report a Problem"|
  • remove unneeded line
sed 's/^Free/\$0.00/'
  • make line structures comparable
sed 's/, v[^,]*//'
  • remove version info
tr -d ","`
  • remove commas
tr -d "*"
  • remove any asterisks in app names
sed 's/ -- /\*/g'
  • put a marker between records
tr "*" "\n"
  • replace marker with newline
grep "Seller"
  • remove records that aren't apps (all apps have sellers)
sed 's/ Seller:/HBPH/'
  • replace Seller with marker for comma insertion
sed 's/ \$\([0-9\.]\{4\}\)/, \1/g'
  • insert comma before price; remove $
sed 's/HBPH/,/g'
  • replace marker with comma
sed 's/^/\"/' | sed 's/$/\"/' | sed 's/, /\", \"/g'
  • put quotes around each field
As a side note, people get huffy about using cat before a command that can simply take a file name as an argument (the "Useless Use of Cat" award). However, I find it useful when I'm working on commands in a modular way. If I need to move the first grep statement around, I don't need to move the file name too.

I analyzed my data, and wow, I've bought a lot of apps. I couldn't bear to leave the left axis on, sorry.

It would be nice to have the date purchased too.

Tuesday, January 25, 2011

Law school:predatory lending ?=science training:pyramid scheme?

The New York Times profiles some unemployed, debt-encumbered lawyers and discusses how law schools report the status of their graduates in a misleading way. These reports promote the perception that going to law school is a good investment.

Catchy quote:
“Enron-type accounting standards have become the norm...Every time I look at this data, I feel dirty.” - law school professor William Henderson of Indiana University
The article reminds me of several recent articles pointing out the crummy conditions for people with science PhDs and how there too the interests of the educating institutions are at odds with those of the applicants and trainees. In the case of science, this is the result of how our academic science research and training is organized: a structure not infrequently called a pyramid scheme.

A faculty person who leads a lab is high on the pyramid, and the majority of the work is done by trainees (postdocs and graduate students) who are lower on the pyramid and mostly want to eventually lead their own lab. This structure was better suited to the expansive early days of civilian science (mid-1900s to 1970ish), because the the whole pyramid was growing, and trainees were relatively more likely to be able to move up the pyramid. When academic research expansion slowed, there were fewer new jobs created. For existing labs to continue, they needed a steady supply of trainees, but there was less demand for those people after they were trained.

What is the outcome? Between 1973 and 1996, the percent of scientists NOT working in permanent, full-time positions rose from 7% to 21%. This measurement was made among scientists who earned their PhDs 9-10 years previously, the assumption being they should no longer be in short-term training positions like a postdoctoral fellow positions.

(This statistic is based on figure 3.12 "Fraction of US life-science PhDs not holding permanent full-time jobs in science or engineering, 1973, 1985, and 1995" in Trends in the Early Careers of Life Scientists. National Research Council (US) Committee on Dimensions, Causes, and Implications of Recent Trends in the Careers of Life Scientists.
Washington (DC): National Academies Press (US); 1998.)

While this seems like an unsustainable system, at least PhD scientists usually don't accumulate hundreds of thousands of dollars in graduate school debt . Faint praise indeed.

Monday, January 17, 2011

Good girl dinette

I ate at Good girl dinette tonight with KC and MM. Yummy Vietnamese food, simple and delicious. House-made sodas, very good spring rolls, good mushroom sandwich. Crummy but friendly service.

Tuesday, January 11, 2011

The market for used iPhones

This beanplot shows the distribution of prices for which used 32GB 3GS iPhones were sold on eBay. A beanplot shows "a density trace, which is mirrored to form a polygon shape." Overlaid is "a one-dimensional scatter plot show[ing] all the individual measurements." When two phones were sold for the same price, the line is twice as wide, and so on.

Here's the R code that produced the plot

allprices= price_and_shipping
$V1+ price_and_shipping$V2
(filename = "priceDistribution.png")
(prices,method="stack",axes=TRUE,what=c(0,1,0,1),log="",main="Beanplot distribution of prices")

code formatting kludged together from output from Pretty R at -- i had to hack at it to get it to work on blogspot, and that made it uglier than it should be. normally they make (as per the name) pretty stuff.

Monday, January 10, 2011

Assessing prices on eBay

I recently listed my old iPhone for sale on eBay. I wanted to know how much phones were going for. But I didn't just want to peruse the prices; I wanted to plot them and analyze them. Here are the steps I took. They are ugly, but they worked on my system, which is Mac OS X Leopard, Firefox 3, and the Table2Clipboard extension (which you don't really need). I'm interested to hear about better ways to do this.

Do an eBay search.
  • search text: iphone 3gs 32
  • Preferences: Completed listings
  • Preferences: Used
  • show 200 results

Copy results to a local file
  • create dedicated directory like /Users/exampleUser/ebaySoldPrice
  • copy results tables only:
  • select a word in the first listing
  • from the right click menu, choose table2clipboard/select table
  • scroll to end of page,
  • shift click in last item to extend selection to include all items,
  • right click and choose View selection source
  • it takes a while for this window to populate
  • Copy, paste and save text as "ebay_search_results.txt"

Clean up the results
  • Open a terminal window and type the following commands (you may have to fix line breaks if you copy this text)

    • cd /Users/exampleUser/ebaySoldPrice/

    • cat ebay_search_results.txt  | sed 's/~//g' | 
      sed 's/<table class="li" r="/~/g' |
      tr "~" "n" | sed 's/ shipping.*$//' |
      sed 's/([0-9]*).*class="vip">/1~/' |
      grep Sold | sed 's/<[^$]*$/~/' |
      sed 's/Free shipping/$0/' | sed 's/<[^$]*$/~/' |
      sed 's/<.*$/~na/' | tr "~" "t"

here's how that breaks down:
  • sed s'/\~//g'
                    ### remove any tildes in ad title
  • tr  "~" "\n"
                    ### replace tildes with newline
  • sed 's/<table class="li" r="/~/g'
                    ### replace table starts with tildes
  • sed 's/ shipping.*$//'
                    ### get rid of text after shipping
  • sed 's/\([0-9]*\).*class\=\"vip\"\>/\1\~/' |
                    ### keep the initial row number, deletes everything else until the title
  • grep Sold
                    ### limit to items that sold
  • sed 's/\<[^$]*\$/~/'  |
                    ### delete text between title and price
  • sed 's/Free shipping/\$0/'
                    ### give free a numerical value
  • sed 's/\<[^$]*\$/~/'  |
                    ### delete text between price and shipping price
  • sed 's/<.*$/~na/'
                    ### if price doesn't have a dollar amount, delete text and give it an NA value
  • tr "~" "\t"
                    ### replace tildes with tabs
  • >ebay_search_results_cleaned.txt
                    ### direct the output into a file

Monday, January 03, 2011

Viet Noodle Restaurant in Atwater Village

photo by seanbonner

I love the spare aesthetic of this restaurant. I ate here today. The service was abysmally slow, but friendly. I had the vegetable soup, which was delicious. MM had the Beef Pho, which he said was fine, but was annoyed that he hadn't remembered it was nothing special.

Rose bowl post-parade float viewing

photos 1,3 and 4 by modmodworld

The day after the parade, I went to see the floats. It hadn't been on my LA bucket list, but it was urged, so I went. The float I liked most reminded me of the pottery design Star Glow (left), and thematically of two other patterns I like a lot: Salem Free Form Hop Scotch Pink and Jack Straw's Constellation (3rd and 4th from left). The photos are by Flickr user modmodworld who has a tremendous collection of photos of mid century dinnerware.