iCloud Reminders in Org-mode: Talking to OS X with Emacs

In this article I implement a library that allows one to synchronize Org-mode to-do items directly to Apple’s iCal. More generally, I describe a synchronization layer between OS X, Clozure Common Lisp, and Emacs, and muse on its potential.

Background

It is a source of joy and pain to customize and extend Emacs. There are many reasons for the attempt, however, and every person has different reasons. I typically describe a few of mine as follows.

  1. When tasks are performed alongside each other in Emacs, it is conceptually simpler to move between them. Most commands have a sort of “Do What I Mean” universality to them.
  2. Passing data between membranes – copying a URL in a web browser to a chat window, e-mailing a snippet of code to a mailing list, or writing an Org-mode outline before publishing it online – is typically effortless.
  3. Every mode and function in Emacs is another tool in your toolbox. You can easily combine tools in the Unix way to slice through work. When you have more tools at your disposal, you can solve more difficult or nuanced problems.

On the other hand, I have an iPhone and an iCloud account, and don’t spend all my time in Emacs exclusively. I want to be able to use the fantastic software outside of Emacs, but have the power and flexibility of Emacs.

I’ve created a dual-language project. One half is in Emacs Lisp; the other in Common Lisp. It uses the foreign function interface provided by Clozure CL. From Emacs, I use SLIME to access Clozure’s REPL, which allows me to make calls to the storage APIs for iCal. Once I retrieve the results I want, I can pass them back to Emacs. I can also make changes using this API, allowing Emacs to inform iCal of what changes have been made in my to-do lists.

Environment Setup

There is a one-time cost of a complex environment setup. You will need a copy of OS X and Emacs. I am doing all this work on Lion, and I don’t know if earlier versions of OS X will have all the tools we need. I am using Emacs 24:

GNU Emacs 24.0.90.1 (i386-apple-darwin11.2.0, NS apple-appkit-1138.23)

I should mention that Emacs 24 is incredibly stable. There is a lot of activity and approval surrounding its use as a day-to-day text editor.

Make sure you install Clozure Common Lisp. Clozure’s Objective-C support is a unique feature of its distribution. This article could not have been written without the indispensable support of their wiki and manual.

You can do this however you’d like, but homebrew makes it easy.

$ brew update
$ brew install clozure-cl

You should have ccl64 in /usr/local/bin, and it should be at least version 1.7. Type (quit) at the REPL to exit it.

You will need SLIME, the essential Lisp IDE for Emacs. What I do is open up my Lisp REPL in my shell, install SLIME through Quicklisp, and then symbolically link the package into my Emacs directory (a tool exists to do this, quicklisp-slime-helper, but out of habit I still do it the more old-fashioned way):

$ ccl64 --load quicklisp.lisp

? (quicklisp-quickstart:install :path ".quicklisp-ccl64/")
? (ql:add-to-init-file)
? (ql:quickload 'swank)
? (quit)

$ ln -s $HOME/.quicklisp-ccl64/dists/quicklisp/software/slime-*-cvs/ $HOME/.emacs.d/vendor/slime

I’ve also built up a bit of a preparation function for initializing SLIME, so I will reproduce that below as well.

(defun quicklisp-slime-setup ()
  (add-to-list 'load-path (expand-file-name "~/.emacs.d/vendor/slime"))
  (add-to-list 'load-path (expand-file-name "~/.emacs.d/vendor/slime/contrib"))
  (setq inferior-lisp-program "/usr/local/bin/ccl64")
  (setq slime-autodoc-mode t)
  (setq slime-net-coding-system 'utf-8-unix)
  (require 'slime)
  (slime-setup '(slime-repl slime-fancy)))

Building ffigen

ffigen is a utility that slurps up frameworks and provides foreign function interfaces into ‘intermediate forms’; in this case, Lisp forms. These instructions exist on the Clozure CL wiki. I’m reproducing them here since they’re not easy to find and the wiki may change.

Building a Leopard version with ObjC 2.0 support

# Note that the svn directory is misnamed; this is actually based on Apple GCC 5464 (not 6465).
$ svn co http://svn.clozure.com/publicsvn/ffigen4/branches/ffigen-apple-gcc-6465/ffigen4

$ cd ffigen4
$ make

$ sudo tar zxvf ffigen-apple-gcc-5465-intel-*.tar.gz -C /usr/local

Generating the Foreign Function Interface

Calendar Store is the name of the framework that Apple provides for accessing calendar data. We need to generate its foreign function interface for Clozure. I did so by copying the FFI generation script from the addressbook framework that Clozure provides access to, and turning it into a script for the calendarstore framework. The script, populate.sh, uses the h-to-ffi.sh terminal command installed by building ffigen above.

I should add that this works with the version of Xcode distributed by the App Store, as well as any version of Xcode you may have in /Developer, but I’m only documenting the former. The changes needed for the latter should be self-evident.

 1: $ export SDK=/Applications/Xcode.app/Contents/Developer/Platforms/\
 2: MacOSX.platform/Developer/SDKs/MacOSX10.6.sdk
 3: 
 4: $ cd $SDK/System/Library/Frameworks/CalendarStore.framework/Headers
 5: 
 6: $ sed '1i\
 7: #import <Foundation/Foundation.h>\
 8: ' CalendarStore.h | sudo tee CalendarStore.ffi.h
 9: 
10: $ cd /usr/local/Cellar/clozure-cl/1.8/ccl/darwin-x86-headers64
11: $ mkdir -p calendarstore/C
12: $ sed 's/AddressBook/CalendarStore/;s/AddressBook/CalendarStore.ffi/' \
13:   addressbook/C/populate.sh > calendarstore/C/populate.sh
14: $ sed -i '' "s/^SDK.*/SDK=$SDK/" calendarstore/C/populate.sh
15: $ chmod +x calendarstore/C/populate.sh
16: $ cd calendarstore/C
17: $ sh populate.sh

Line 1 above assumes that you are using Xcode off of the App Store. If not, the SDKs/ directory will be located in /Developer.

Lines 6, 7, and 8 comprise a single command, adding a missing import to the CalendarStore header file. Once we populate the framework, we must then tell Clozure about it.

CL-USER> (require 'parse-ffi)
CL-USER> (ccl::parse-standard-ffi-files :calendarstore)

After this is done, we never have to do it again. From now on, whenever we open up a Clozure REPL, we can just load up everything we need.

CL-USER> (require 'objc-support)
CL-USER> (objc:load-framework "CalendarStore" :calendarstore)

Explaining the Bridge

There are a couple aspects of Objective-C and how they translate through the Bridge that we need to be aware of.

Methods in Objective-C

Objective-C uses a message passing syntax. Whereas languages like C++ bind method names to some code during compilation, Objective-C resolves message receivers during runtime. Message names are typically longer than method names because they also contain identifying information about the message’s parameters, separated by colons.

Whereas in C++ you may call something like:

obj->method(foo, bar)

In Objective-C you would call:

[obj method:foo andBar:bar]

And we’d consider the message name to be method:andBar:. Yes, even the colons are part of the message name. In our bridge, that method invocation would become:

(#/method:andBar: obj foo bar)

Now that we know this, we can start using Apple’s API to pull objects out of the runtime. Open up a REPL with Clozure CL as your inferior Lisp. First we need to load up the bridge.

CL-USER> (require 'objc-support)

Many commonly used Foundation Framework objects are available for us immediately. These objects start with the prefix ns-. An NSDate would become an ns-date. An NSString would become an ns-string. Say we wanted to create an NSDate with the current date and time, and print its string representation. Let’s put this functionality in a function called today_as_string.

Take a look at the difference as to how methods are called in Objective-C. today_as_string, implemented in Objective-C, and then Lisp.

NSString* today_as_string() {
  NSDate *today = [NSDate date];
  NSString *date_string = [today description];
  [today release];
  return date_string;
}

(defun today-as-string ()
  (let* ((today (#/date ns:ns-date))
         (date-string (#/description today)))
    (#/release today)
    date-string))

Memory Management

Based on the Clozure documentation (§15.3.3), an autorelease pool is created for us in the toplevel. This means as good citizens, we should manually release the memory when we finish using it. Calling [NSDate date] automatically calls init and alloc on the instance, and the (#/release today) call above dovetails that by decreasing the reference count.

For more detail on reference counting, please see Apple’s “Advanced Memory Management Programming Guide”.

Miscellaneous

Here are some other things to note. We can get a reference to any class by passing its camel-cased name to @class.

CL-USER> (objc:@class "NSString")
#<OBJC:OBJC-CLASS NS:NS-STRING (#x7FFF75FE69F8)>

We can determine if an object is a null pointer by using %null-ptr-p:

CL-USER> (%null-ptr-p (#/date ns:ns-date))
NIL
CL-USER> (%null-ptr-p (%null-ptr))
T

We are given a reader macro for generating NSStrings (prefix it with a pound sign, #), and we can get a Lisp string out of an NSString with Clozure’s lisp-string-from-nsstring function:

CL-USER> (ccl::lisp-string-from-nsstring #@"test")
"test"

The use of the pound sign in the above code ((#/date ns:ns-date)) is a reader macro as well. You can get a hint as to how it works by checking the contents of the package nextstep-functions.

Introducing nsclasp

I’ve written a small utility library for Common Lisp called nsclasp. It provides a set of functions that are very useful when working with Apple’s Foundation Framework. You can retrieve it by cloning it into your Quicklisp installation’s local-projects directory and refreshing the project list:

$ cd ~/.quicklisp-ccl64/local-projects
$ git clone git://github.com/ardekantur/nsclasp.git
CL-USER> (ql:register-local-projects)
CL-USER> (ql:quickload 'nsclasp)

For more information, see the project documentation.

Implementation

The most important aspect of iCalSync, the thing that makes this so delightfully easy, is that Emacs Lisp can talk directly to the Clozure instance. The key function is this:

(defun icalsync-capture-ccl-result
(command)
  "Convert a result from the SLIME Lisp's REPL into the Emacs\nLisp toplevel."
  (read
   (slime-eval
    `(swank::pprint-eval ,command))))

Using this, we can send code to Clozure’s REPL, get the result back, and turn it into a Lisp form.

First Steps: Retrieving Calendar Names

Let’s make two useful functions for accessing the CalendarStore, then try to retrieve a list of calendar names from Clozure.

(defun icalsync-cl-calendar-store ()
  "Return the calendar store. Equivalent to
    [CalCalendarStore defaultCalendarStore]."
  (#/defaultCalendarStore ns:cal-calendar-store))

(defun icalsync-cl-all-calendars ()
  "Return a list of all calendars in the default calendar
    store. Equivalent to [[CalCalendarStore defaultCalendarStore]
    calendars]."
  (#/calendars (icalsync-cl-calendar-store)))

(let ((calendars (icalsync-cl-all-calendars)))
  (loop for i upto (1- (#/count calendars))
     for calendar = (#/objectAtIndex: calendars i)
     do (format t "~A, " (ccl::lisp-string-from-nsstring (#/title calendar)))))
Calendar, Matthew Snyder, Do, Buy, Inbox, Example, 

Our handling of the NSArray is a little clumsy. We can’t just mapcar over it, we need to explicitly iterate over its elements by index. That’s why nsclasp is helpful, as it contains a method to make this a little nicer:

(let* ((calendars (nsclasp:ns-array->list (icalsync-cl-all-calendars))))
  (format t "~{~a, ~}" (mapcar (lambda (c) (ccl::lisp-string-from-nsstring (#/title c))) calendars)))

But no matter how you do it, you should have been able to retrieve all your calendar names. As it turns out, this is an important piece of information to have, as the Foundation Framework doesn’t yet allow us to create the kind of calendar that would sync with iCloud for us. For the time being, we have to create the calendar we want our TODOs synced to, in iCal.

new-calendar-list.png

First Steps: Creating a CalTask

Now let’s combine this on the tail end of creating an iCal reminder, and sending it to a calendar.

Tasks exist in iCal as instances of the CalTask object of the CalendarStore Framework. Every task has a title, UID, a parent calendar, alarms, due dates, completed dates, notes, and other such attributes. We have to describe a mapping. Our input function takes an Emacs buffer and returns a list of parsed TODOs, and what calendar they should be synced to.

(defun icalsync-el-transform-todos (buffer)
  "Turn an org mode buffer into a list of iCal tasks.

A function that describes one way to retrieve all of the valid
TODOs from a buffer for turning into respective iCal tasks. You
can write a function that best suits your workflow. The function
is expected to take the name of a buffer, or a buffer, and return
a list of plists, each one representing a TODO item / iCal task.

FIXME(msnyder): Use org-map-entries to make this easier."
  (save-excursion
   (let ((calendar-uid (icalsync-el-calendar-name)) todos)
     (switch-to-buffer buffer)
     (beginning-of-buffer)
     (while (re-search-forward org-todo-line-regexp nil t)
      (let* ((todo-text (match-string-no-properties 3))
             (components (org-heading-components))
             (todo-state (nth 2 components))
             (headline (nth 4 components))
             (scheduled-time (org-get-scheduled-time (point)))
             todo-uid)
        (if (and todo-state
                 (funcall icalsync-el-valid-p-function todo-state todo-text))
            (progn (if (string-match org-todo-line-tags-regexp headline)
                       (setq headline (nth 4 (org-heading-components)))
                       (setq todos
                             (cons (list :buffer
                                         (buffer-file-name)
                                         :point
                                         (point)
                                         :calendar-uid
                                         calendar-uid
                                         :incomplete-p
                                         (funcall
                                          icalsync-el-incomplete-p-function
                                          todo-state)
                                         :title
                                         (org-trim headline)
                                         :scheduled
                                         (and
                                          scheduled-time
                                          (decode-time scheduled-time))
                                         :uid
                                         (cdr
                                          (assoc
                                           "ICALSYNC-UID"
                                           (org-entry-properties)))
                                         :dt
                                         (and
                                          (cdr
                                           (assoc
                                            "ICALSYNC-DT"
                                            (org-entry-properties)))
                                          (read
                                           (cdr
                                            (assoc
                                             "ICALSYNC-DT"
                                             (org-entry-properties))))))
                                   todos)))))))
     todos)))

I’ve created a basic Org file.

#+TITLE: example.org

* TODO Recycling
,  SCHEDULED: <2012-04-15 Sun>
* TODO Trash
* TODO Fix toilet

I run the function on it, and after getting asked which calendar I want the items to belong to, I get the following list of lists.

(icalsync-el-transform-todos "example.org")
:buffer /…/example.org :point 78 :calendar-uid C854CB60 :incomplete-p t :title Fix toilet :scheduled nil :uid nil :dt nil
:buffer /…/example.org :point 60 :calendar-uid C854CB60 :incomplete-p t :title Trash :scheduled nil :uid nil :dt nil
:buffer /…/example.org :point 17 :calendar-uid C854CB60 :incomplete-p t :title Recycling :scheduled (0 0 0 15 4 2012 0 t -14400) :uid nil :dt nil

When we upload a task into the Calendar Store, we’re going to provide its title, deadline, and whether or not it’s complete. There’s also priority, which would be nice, but not necessary for the time being. Once the upload is successful, we’re going to also retrieve the task’s UID, and its dateStamp property, and save them in the property drawer of the TODO, in order to make synchronization easier.

(defun icalsync-el-upload-todo (todo)
  (let (todo-uid)
    (save-excursion
     (setq todo-uid
           (icalsync-capture-ccl-result
            (format "(icalsync-cl-create-task '%S)" todo)))
     (find-file (getf todo :buffer)) (goto-char (getf todo :point))
     (goto-char (car (org-get-property-block nil nil t)))
     (org-set-property "ICALSYNC-UID" todo-uid)
     (org-set-property "ICALSYNC-DT" (current-time)))))

(defun icalsync-el-upload-todos (todos)
  (loop for todo in todos do (icalsync-el-upload-todo todo)))

From here, all we need to do is run the Emacs Lisp functions needed to transform all the TODOs and upload them into the Calendar Store. Keep iCal open when you do this and make sure the Reminder list is visible.

shown-calendar-list.png

(setq *todos* (icalsync-el-transform-todos "example.org"))
(icalsync-el-upload-todos *todos*)

Et voila!

new-reminders.png

First Steps: Retrieving Changes

So now you have your Org-mode tasks in a Reminder list on your iPhone. Brilliant! You wander around the town, happily marking off the items you’ve completed. How can we make Org-mode aware of the items that were checked off?

This is a very rudimentary method, but it works. What we’ll do is update TODOs based on buffer, the same way we uploaded them to iCal. For a given buffer, we’ll walk through the TODOs, seeing if they have a UID handed out by the calendar store. If so, we ask the calendar store about it. We compare the time-stamp of the TODO to the time-stamp of the reminder, and if the reminder has been changed more recently, we update the TODO.

(defun icalsync-el-download-todo (todo)
  (save-excursion (find-file (getf todo :buffer))
   (goto-char (getf todo :point))
   (let* ((result
           (icalsync-capture-ccl-result
            (format "(icalsync-cl-task->plist (icalsync-cl-get-task %S))"
                    (getf todo :uid))))
          (t1
           (apply 'encode-time
                  (read (cdr (assoc "ICALSYNC-DT" (org-entry-properties))))))
          (t2 (apply 'encode-time (getf result 'datestamp))))
     (if (time-less-p t1 t2)
         (progn (if (getf result 'completeddate) (org-todo 'done))
                (org-set-property "ICALSYNC-DT" (format "%S" t2)))))))

Not the use of the function icalsync-cl-task->plist. Since we can’t hand off a CalTask object from Clozure to Emacs Lisp, I’ve written a function that distills a task down to its most important parts.

(defvar *task-properties* (list "title"
                                "uid"
                                "priority"
                                "dueDate"
                                "dateStamp"
                                "completedDate"))

(defun icalsync-cl-task->plist (task)
  "Convert a CalTask to a plist of properties. The list of properties
    is described by *task-properties*."
  (loop for property in *task-properties*
        for value = (funcall (intern property "NEXTSTEP-FUNCTIONS") task)
        append (list (intern (format nil "~:@(~a~)" property))
                     (icalsync-cl-coerce-value value))))

Next Steps

There are a few things missing from this implementation. We don’t check for the existence of a task before creating it from the corresponding TODO. Priority is not synced. Org-mode is not made aware of items added to Reminders.app. We track where our TODOs are based on the buffer name and their position in the buffer, which will quite easily break if we shuffle our TODOs around. Only the reminder’s completion status is synced back to Emacs, but if its description or due date changes, we should synchronize those as well. As a proof-of-concept, however, I think this is a solid start.

In addition, the behavior that you use to turn Org-mode files into lists of reminders – or vice versa – is entirely up to you. There are countless ways to do so. A conversion could, for example, take each calendar, and make it a headline in an Org file, with each task becoming a checklist item with a property drawer. Another conversion could check the TODOs for a custom condition, i.e., only tell iCal about TODOs which have a certain property in their drawer, or are scheduled after a certain date.

All the code written for this article is available in my elstar GitHub repository. The elstar name for my repository is a bannerhead of sorts, an invitation for this method to be explored further, and to collect the results in a shared place. I welcome any contributions.

Conclusion

The point of this article was not just to create a synchronization layer for iCal to Org-mode, but to explore the method of synchronization itself. For Emacs users on OS X, the potential exists to fill in the gaps with their usage of operating system (and associated mobile/cloud convenience) and text editor. Some other ideas that may be worth pursuing:

  • An Emacs mode for interfacing with iTunes
  • Functionality to convert Address Book entries into BBDB or org-contacts entries
  • Turning events on calendars into Emacs diary entries

I look forward to seeing what some enterprising Lisp hackers can come up with.