Josh Haas's Web Log

Per my previous post on making the world a more joyful place, I’m interested in self-experimentation. There are a lot of cool things that open up if you can take a scientific approach to your own life, but in the past I haven’t been able to follow through because I lacked the discipline to record data regularly.

What I really want is for data collection to be totally mindless. My ideal is, I get a text message asking me the question, and all I have to do is reply to the message. This is really easy for me since I generally reply to text messages in real time.

Setting something like that up used to be a lot of work. But thanks to a new free service, if this then that, I was able to super-easily set up a simple experiment where I get texted a yes / no question at 5 random times each day, and have the data be automatically collected for me! Here’s a walk-through of how I did it. You’ll need: the internet, an ifttt account, a gmail account, an evernote account, and a webserver where you can publish python scripts and run cron jobs (I use dreamhost — it’s super-cheap shared hosting, perfect for lightweight little things like this).

Prelude: ifttt basics.

ifttt lets you create dead-simple automated tasks by connecting “triggers” (which can be anything from you sending ifttt a email or text message, to you posting on facebook or twitter), to “actions” (which can be ifttt sending you an email or text message, or posting something to a 3rd party account). It integrates with pretty much every popular webservice on the internet, and it’s almost so self-explanatory that it hurts:

My ifttt home page showing the three tasks I created for this experiment

Step 1: Generate the reminders.

If you’re okay with setting the exact time you get the message, this part is pretty much done: go into ifttt, and create a task with a “Date & Time” trigger and an SMS action.

For me, I wanted to add some randomness to when I get the reminders, so I had to do a little more work. On my web server, I created a script that, when run, sends an email from my gmail account to ifttt. I created a file “email_google.py” with the following contents:

import os
import smtplib
import mimetypes
from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email.MIMEAudio import MIMEAudio
from email.MIMEImage import MIMEImage
from email.Encoders import encode_base64
import base64

def send_mail(recipient, subject, text):
    user = 'fake@gmail.com'
    pw = 'crazy_random_fake'
    msg = MIMEMultipart()
    msg['From'] = user
    msg['To'] = recipient
    msg['Subject'] = subject
    msg.attach(MIMEText(text))
    mailServer = smtplib.SMTP('smtp.gmail.com', 587)
    mailServer.ehlo()
    mailServer.starttls()
    mailServer.ehlo()
    mailServer.login(user, pw)
    mailServer.sendmail(user, recipient, msg.as_string())
    mailServer.close()
    print('Sent email to %s' % recipient)

And then another file “send_reminder.py” that does the actual work:

import random
import email_google
import time

time.sleep(random.randint(0, 110) * 60) #subtracting 10 minutes to account for ifttt being slow

email_google.send_mail('trigger@ifttt.com', '#myexperiment', 'body')

What this does is sends an email to ifttt anywhere from 0 to 110 minutes after the script is started. I then run this script using a cronjob (“python /path/to/script/send_reminder.py”) — dreamhost’s control panel provides a nice gui for setting up cron jobs so I didn’t have to do this by hand.

Then in ifttt, I set up a task with an email trigger with the hashtag #myexperiment, and an SMS action that sends me the reminder. So whenever the script on my server gets kicked off, it waits a few minutes, then emails ifttt, which sends me a text message!

Step 2: Capture the reply.

I want to be able to reply back to the text message and have that information saved in a database. First, I created a database on my webserver using the sqlite3 command: I ssh’ed in and ran “sqlite3 my_db”. That pops up the sqlite3 command line tool; I ran “create table log (time, input);” to create a table to store the data, and then “.exit” to leave sqlite3.

I then created a simple script that stores a one-word response to the database. I created the file “accept.py” with the following contents:

#!/usr/bin/python

print "Content-type: text/html\n\n"

import cgi
import cgitb
cgitb.enable()

import sqlite3


form = cgi.FieldStorage()
input = form["input"].value.lower()

if input in ('yes', 'no'):
    con = sqlite3.connect('my_db')
    con.execute('insert into log select datetime(), ?', [input])
    con.commit()

print 'success'

This file goes on my web server. I needed to run “chmod 755 accept_input.py” to let Apache run it as a script. What this does: when I go to http://myserver.com/path/to/my/folder/accept.py?input=yes, it saves “yes” to my database, and records the time at which it happened. Likewise, changing the url to “?input=no” saves “no”.

Finally, I need to make ifttt visit that url when I send it a text message. I created a task with an SMS trigger. But what should be the action? Right now ifttt doesn’t have a “visit url” action. However, we can fake it by using the Evernote channel’s “create a link note” channel, which creates a link in your notebook based off of a given url. Evernote visits the url to create the note, so the server registers the hit. We need to get the “yes” or “no” message we send into the url, so I set the “Link URL” field of the action to “http://myserver.com/path/to/my/folder/accept.py?input={{Message}}”.

Voila! Every time we text a reply to ifttt, the body of the reply (a “yes” or “no”) gets stored in our database!

Step 3. Review the results.

I want to see the results of the experiment! For me, the thing I want to track is the percentage of “yes”s for a given day and how that changes over time. So, I wrote one more python script that reads the database and outputs a table:

#!/usr/bin/python

print "Content-type: text/html\n\n"

import cgi
import cgitb
cgitb.enable()

import sqlite3


con = sqlite3.connect('my_db')
results = con.execute("""
    select date(time), (sum(case when input = 'yes' then 1 else 0 end) +0.0) / count(*) 
    from log 
    group by date(time); """)

print """
<html>
<head>
<style>
td {
    padding: 3px;
}
table, tr, td {
    border: 1px solid black;
}
</style>
</head>

<body><table>

"""


for row in results:
    print "<tr><td>", row[0], "</td><td>", row[1], "</td></tr>"
    
print '</table></body></html>'

Save that to your webserver, and remember to run “chmod 755” on it if necessary! Now, you can visit the url of that script and see a nice little report on your experiment.

Bonus step: Backing things up

I’m paranoid, and because dreamhost is cheap shared web-hosting (no offence, guys), I’m not 100% sure a server crash won’t annihilate all my hard-won data! So I set up one more ifttt task to back it up. I used the “Date & Time” trigger, setting it to run once a month. For the action, I told it to save a given url to my Dropbox account which is what I use for all my personal backups. What url? The url of the database… sqlite3 saves its data as a simple file, so when we ran “sqlite3 my_db” above, it created a file called my_db. Since it is in the same folder as your scripts, you can just point your browser to “http://myserver.com/path/to/my/folder/my_db” and it will download the latest version of the database. Or, in our case, have ifttt download it automatically and save it to your dropbox!

Conclusion:

I’m falling in love with ifttt. There’s a lot of stuff that I had to do manually with python and cron jobs, but ifttt is new and I can imagine as they improve their service, more and more of the steps above will be able to be completely automated. In the past, setting something like this up would have been a big pain (although I hear twilio makes interacting with text messages reasonably easy)… but we live in the future, ladies and gentlemen!