iCloud Reminders in Org-mode: Talking to OS X with Emacs
In this article I implement a library that allows one to synchronize Org-mode to-do items directly to Apple's iCal. More generally, I describe a synchronization layer between OS X, Clozure Common Lisp, and Emacs, and muse on its potential.
Table of Contents
Background
It is a source of joy and pain to customize and extend Emacs. There are many reasons for the attempt, however, and every person has different reasons. I typically describe a few of mine as follows.
- When tasks are performed alongside each other in Emacs, it is conceptually simpler to move between them. Most commands have a sort of "Do What I Mean" universality to them.
- Passing data between membranes – copying a URL in a web browser to a chat window, e-mailing a snippet of code to a mailing list, or writing an Org-mode outline before publishing it online – is typically effortless.
- Every mode and function in Emacs is another tool in your toolbox. You can easily combine tools in the Unix way to slice through work. When you have more tools at your disposal, you can solve more difficult or nuanced problems.
On the other hand, I have an iPhone and an iCloud account, and don't spend all my time in Emacs exclusively. I want to be able to use the fantastic software outside of Emacs, but have the power and flexibility of Emacs.
I've created a dual-language project. One half is in Emacs Lisp; the other in Common Lisp. It uses the foreign function interface provided by Clozure CL. From Emacs, I use SLIME to access Clozure's REPL, which allows me to make calls to the storage APIs for iCal. Once I retrieve the results I want, I can pass them back to Emacs. I can also make changes using this API, allowing Emacs to inform iCal of what changes have been made in my to-do lists.
Environment Setup
There is a one-time cost of a complex environment setup. You will need a copy of OS X and Emacs. I am doing all this work on Lion, and I don't know if earlier versions of OS X will have all the tools we need. I am using Emacs 24:
GNU Emacs 24.0.90.1 (i386-apple-darwin11.2.0, NS apple-appkit-1138.23)
I should mention that Emacs 24 is incredibly stable. There is a lot of activity and approval surrounding its use as a day-to-day text editor.
Make sure you install Clozure Common Lisp. Clozure's Objective-C support is a unique feature of its distribution. This article could not have been written without the indispensable support of their wiki and manual.
You can do this however you'd like, but homebrew makes it easy.
$ brew update $ brew install clozure-cl
You should have ccl64 in /usr/local/bin, and it should be at
least version 1.7. Type (quit) at the REPL to exit it.
You will need SLIME, the essential Lisp IDE for Emacs. What I do is open up my Lisp REPL in my shell, install SLIME through Quicklisp, and then symbolically link the package into my Emacs directory (a tool exists to do this, quicklisp-slime-helper, but out of habit I still do it the more old-fashioned way):
$ ccl64 --load quicklisp.lisp ? (quicklisp-quickstart:install :path ".quicklisp-ccl64/") ? (ql:add-to-init-file) ? (ql:quickload 'swank) ? (quit) $ ln -s $HOME/.quicklisp-ccl64/dists/quicklisp/software/slime-*-cvs/ $HOME/.emacs.d/vendor/slime
I've also built up a bit of a preparation function for initializing SLIME, so I will reproduce that below as well.
(defun quicklisp-slime-setup () (add-to-list 'load-path (expand-file-name "~/.emacs.d/vendor/slime")) (add-to-list 'load-path (expand-file-name "~/.emacs.d/vendor/slime/contrib")) (setq inferior-lisp-program "/usr/local/bin/ccl64") (setq slime-autodoc-mode t) (setq slime-net-coding-system 'utf-8-unix) (require 'slime) (slime-setup '(slime-repl slime-fancy)))
Building ffigen
ffigen is a utility that slurps up frameworks and provides foreign
function interfaces into 'intermediate forms'; in this case, Lisp
forms. These instructions exist on the Clozure CL wiki. I'm
reproducing them here since they're not easy to find and the wiki may change.
Building a Leopard version with ObjC 2.0 support# Note that the svn directory is misnamed; this is actually based on Apple GCC 5464 (not 6465). $ svn co http://svn.clozure.com/publicsvn/ffigen4/branches/ffigen-apple-gcc-6465/ffigen4 $ cd ffigen4 $ make $ sudo tar zxvf ffigen-apple-gcc-5465-intel-*.tar.gz -C /usr/local
Generating the Foreign Function Interface
Calendar Store is the name of the framework that Apple provides for
accessing calendar data. We need to generate its foreign function
interface for Clozure. I did so by copying the FFI generation script
from the addressbook framework that Clozure provides access to, and
turning it into a script for the calendarstore framework. The
script, populate.sh, uses the h-to-ffi.sh terminal command
installed by building ffigen above.
I should add that this works with the version of Xcode distributed by
the App Store, as well as any version of Xcode you may have in
/Developer, but I'm only documenting the former. The changes needed
for the latter should be self-evident.
1: $ export SDK=/Applications/Xcode.app/Contents/Developer/Platforms/\ 2: MacOSX.platform/Developer/SDKs/MacOSX10.6.sdk 3: 4: $ cd $SDK/System/Library/Frameworks/CalendarStore.framework/Headers 5: 6: $ sed '1i\ 7: #import <Foundation/Foundation.h>\ 8: ' CalendarStore.h | sudo tee CalendarStore.ffi.h 9: 10: $ cd /usr/local/Cellar/clozure-cl/1.8/ccl/darwin-x86-headers64 11: $ mkdir -p calendarstore/C 12: $ sed 's/AddressBook/CalendarStore/;s/AddressBook/CalendarStore.ffi/' \ 13: addressbook/C/populate.sh > calendarstore/C/populate.sh 14: $ sed -i '' "s/^SDK.*/SDK=$SDK/" calendarstore/C/populate.sh 15: $ chmod +x calendarstore/C/populate.sh 16: $ cd calendarstore/C 17: $ sh populate.sh
Line 1 above assumes that you are using Xcode off of the App
Store. If not, the SDKs/ directory will be located in /Developer.
Lines 6, 7, and 8 comprise a single command, adding a missing import to the CalendarStore header file. Once we populate the framework, we must then tell Clozure about it.
CL-USER> (require 'parse-ffi) CL-USER> (ccl::parse-standard-ffi-files :calendarstore)
After this is done, we never have to do it again. From now on, whenever we open up a Clozure REPL, we can just load up everything we need.
CL-USER> (require 'objc-support) CL-USER> (objc:load-framework "CalendarStore" :calendarstore)
Explaining the Bridge
There are a couple aspects of Objective-C and how they translate through the Bridge that we need to be aware of.
Methods in Objective-C
Objective-C uses a message passing syntax. Whereas languages like C++ bind method names to some code during compilation, Objective-C resolves message receivers during runtime. Message names are typically longer than method names because they also contain identifying information about the message's parameters, separated by colons.
Whereas in C++ you may call something like:
obj->method(foo, bar)
In Objective-C you would call:
[obj method:foo andBar:bar]
And we'd consider the message name to be method:andBar:. Yes, even
the colons are part of the message name. In our bridge, that method
invocation would become:
(#/method:andBar: obj foo bar)
Now that we know this, we can start using Apple's API to pull objects out of the runtime. Open up a REPL with Clozure CL as your inferior Lisp. First we need to load up the bridge.
CL-USER> (require 'objc-support)
Many commonly used Foundation Framework objects are available for us
immediately. These objects start with the prefix ns-. An NSDate
would become an ns-date. An NSString would become an ns-string.
Say we wanted to create an NSDate with the current date and time, and
print its string representation. Let's put this functionality in a
function called today_as_string.
Take a look at the difference as to how methods are called in
Objective-C. today_as_string, implemented in Objective-C, and then
Lisp.
NSString* today_as_string() { NSDate *today = [NSDate date]; NSString *date_string = [today description]; [today release]; return date_string; }
(defun today-as-string () (let* ((today (#/date ns:ns-date)) (date-string (#/description today))) (#/release today) date-string))
Memory Management
Based on the Clozure documentation (ยง15.3.3), an autorelease pool is
created for us in the toplevel. This means as good citizens, we should
manually release the memory when we finish using it. Calling [NSDate date] automatically calls init and alloc on the instance, and the
(#/release today) call above dovetails that by decreasing the
reference count.
For more detail on reference counting, please see Apple's "Advanced Memory Management Programming Guide".
Miscellaneous
Here are some other things to note. We can get a reference to any
class by passing its camel-cased name to @class.
CL-USER> (objc:@class "NSString")
#<OBJC:OBJC-CLASS NS:NS-STRING (#x7FFF75FE69F8)>
We can determine if an object is a null pointer by using
%null-ptr-p:
CL-USER> (%null-ptr-p (#/date ns:ns-date)) NIL CL-USER> (%null-ptr-p (%null-ptr)) T
We are given a reader macro for generating NSStrings (prefix it with a
pound sign, #), and we can get a Lisp string out of an NSString with
Clozure's lisp-string-from-nsstring function:
CL-USER> (ccl::lisp-string-from-nsstring #@"test") "test"
The use of the pound sign in the above code ((#/date ns:ns-date)) is
a reader macro as well. You can get a hint as to how it works by
checking the contents of the package nextstep-functions.
Introducing nsclasp
I've written a small utility library for Common Lisp called
nsclasp. It provides a set of functions that are very useful when
working with Apple's Foundation Framework. You can retrieve it by
cloning it into your Quicklisp installation's local-projects
directory and refreshing the project list:
$ cd ~/.quicklisp-ccl64/local-projects $ git clone git://github.com/ardekantur/nsclasp.git
CL-USER> (ql:register-local-projects) CL-USER> (ql:quickload 'nsclasp)
For more information, see the project documentation.
Implementation
The most important aspect of iCalSync, the thing that makes this so delightfully easy, is that Emacs Lisp can talk directly to the Clozure instance. The key function is this:
(defun icalsync-capture-ccl-result (command) "Convert a result from the SLIME Lisp's REPL into the Emacs\nLisp toplevel." (read (slime-eval `(swank::pprint-eval ,command))))
Using this, we can send code to Clozure's REPL, get the result back, and turn it into a Lisp form.
First Steps: Retrieving Calendar Names
Let's make two useful functions for accessing the CalendarStore,
then try to retrieve a list of calendar names from Clozure.
(defun icalsync-cl-calendar-store () "Return the calendar store. Equivalent to [CalCalendarStore defaultCalendarStore]." (#/defaultCalendarStore ns:cal-calendar-store)) (defun icalsync-cl-all-calendars () "Return a list of all calendars in the default calendar store. Equivalent to [[CalCalendarStore defaultCalendarStore] calendars]." (#/calendars (icalsync-cl-calendar-store))) (let ((calendars (icalsync-cl-all-calendars))) (loop for i upto (1- (#/count calendars)) for calendar = (#/objectAtIndex: calendars i) do (format t "~A, " (ccl::lisp-string-from-nsstring (#/title calendar)))))
Calendar, Matthew Snyder, Do, Buy, Inbox, Example,
Our handling of the NSArray is a little clumsy. We can't just mapcar
over it, we need to explicitly iterate over its elements by
index. That's why nsclasp is helpful, as it contains a method to make
this a little nicer:
(let* ((calendars (nsclasp:ns-array->list (icalsync-cl-all-calendars)))) (format t "~{~a, ~}" (mapcar (lambda (c) (ccl::lisp-string-from-nsstring (#/title c))) calendars)))
But no matter how you do it, you should have been able to retrieve all your calendar names. As it turns out, this is an important piece of information to have, as the Foundation Framework doesn't yet allow us to create the kind of calendar that would sync with iCloud for us. For the time being, we have to create the calendar we want our TODOs synced to, in iCal.
First Steps: Creating a CalTask
Now let's combine this on the tail end of creating an iCal reminder, and sending it to a calendar.
Tasks exist in iCal as instances of the CalTask object of the
CalendarStore Framework. Every task has a title, UID, a parent
calendar, alarms, due dates, completed dates, notes, and other such
attributes. We have to describe a mapping. Our input function takes an
Emacs buffer and returns a list of parsed TODOs, and what calendar
they should be synced to.
(defun icalsync-el-transform-todos (buffer) "Turn an org mode buffer into a list of iCal tasks. A function that describes one way to retrieve all of the valid TODOs from a buffer for turning into respective iCal tasks. You can write a function that best suits your workflow. The function is expected to take the name of a buffer, or a buffer, and return a list of plists, each one representing a TODO item / iCal task. FIXME(msnyder): Use org-map-entries to make this easier." (save-excursion (let ((calendar-uid (icalsync-el-calendar-name)) todos) (switch-to-buffer buffer) (beginning-of-buffer) (while (re-search-forward org-todo-line-regexp nil t) (let* ((todo-text (match-string-no-properties 3)) (components (org-heading-components)) (todo-state (nth 2 components)) (headline (nth 4 components)) (scheduled-time (org-get-scheduled-time (point))) todo-uid) (if (and todo-state (funcall icalsync-el-valid-p-function todo-state todo-text)) (progn (if (string-match org-todo-line-tags-regexp headline) (setq headline (nth 4 (org-heading-components))) (setq todos (cons (list :buffer (buffer-file-name) :point (point) :calendar-uid calendar-uid :incomplete-p (funcall icalsync-el-incomplete-p-function todo-state) :title (org-trim headline) :scheduled (and scheduled-time (decode-time scheduled-time)) :uid (cdr (assoc "ICALSYNC-UID" (org-entry-properties))) :dt (and (cdr (assoc "ICALSYNC-DT" (org-entry-properties))) (read (cdr (assoc "ICALSYNC-DT" (org-entry-properties)))))) todos))))))) todos)))
I've created a basic Org file.
#+TITLE: example.org * TODO Recycling SCHEDULED: <2012-04-15 Sun> * TODO Trash * TODO Fix toilet
I run the function on it, and after getting asked which calendar I want the items to belong to, I get the following list of lists.
(icalsync-el-transform-todos "example.org")
| :buffer | /…/example.org | :point | 78 | :calendar-uid | C854CB60… | :incomplete-p | t | :title | Fix toilet | :scheduled | nil | :uid | nil | :dt | nil |
| :buffer | /…/example.org | :point | 60 | :calendar-uid | C854CB60… | :incomplete-p | t | :title | Trash | :scheduled | nil | :uid | nil | :dt | nil |
| :buffer | /…/example.org | :point | 17 | :calendar-uid | C854CB60… | :incomplete-p | t | :title | Recycling | :scheduled | (0 0 0 15 4 2012 0 t -14400) | :uid | nil | :dt | nil |
When we upload a task into the Calendar Store, we're going to provide
its title, deadline, and whether or not it's complete. There's also
priority, which would be nice, but not necessary for the time
being. Once the upload is successful, we're going to also retrieve the
task's UID, and its dateStamp property, and save them in the
property drawer of the TODO, in order to make synchronization easier.
(defun icalsync-el-upload-todo (todo) (let (todo-uid) (save-excursion (setq todo-uid (icalsync-capture-ccl-result (format "(icalsync-cl-create-task '%S)" todo))) (find-file (getf todo :buffer)) (goto-char (getf todo :point)) (goto-char (car (org-get-property-block nil nil t))) (org-set-property "ICALSYNC-UID" todo-uid) (org-set-property "ICALSYNC-DT" (current-time))))) (defun icalsync-el-upload-todos (todos) (loop for todo in todos do (icalsync-el-upload-todo todo)))
From here, all we need to do is run the Emacs Lisp functions needed to transform all the TODOs and upload them into the Calendar Store. Keep iCal open when you do this and make sure the Reminder list is visible.
(setq *todos* (icalsync-el-transform-todos "example.org"))
(icalsync-el-upload-todos *todos*)
Et voila!
First Steps: Retrieving Changes
So now you have your Org-mode tasks in a Reminder list on your iPhone. Brilliant! You wander around the town, happily marking off the items you've completed. How can we make Org-mode aware of the items that were checked off?
This is a very rudimentary method, but it works. What we'll do is update TODOs based on buffer, the same way we uploaded them to iCal. For a given buffer, we'll walk through the TODOs, seeing if they have a UID handed out by the calendar store. If so, we ask the calendar store about it. We compare the time-stamp of the TODO to the time-stamp of the reminder, and if the reminder has been changed more recently, we update the TODO.
(defun icalsync-el-download-todo (todo) (save-excursion (find-file (getf todo :buffer)) (goto-char (getf todo :point)) (let* ((result (icalsync-capture-ccl-result (format "(icalsync-cl-task->plist (icalsync-cl-get-task %S))" (getf todo :uid)))) (t1 (apply 'encode-time (read (cdr (assoc "ICALSYNC-DT" (org-entry-properties)))))) (t2 (apply 'encode-time (getf result 'datestamp)))) (if (time-less-p t1 t2) (progn (if (getf result 'completeddate) (org-todo 'done)) (org-set-property "ICALSYNC-DT" (format "%S" t2)))))))
Not the use of the function icalsync-cl-task->plist. Since we can't
hand off a CalTask object from Clozure to Emacs Lisp, I've written a
function that distills a task down to its most important parts.
(defvar *task-properties* (list "title" "uid" "priority" "dueDate" "dateStamp" "completedDate")) (defun icalsync-cl-task->plist (task) "Convert a CalTask to a plist of properties. The list of properties is described by *task-properties*." (loop for property in *task-properties* for value = (funcall (intern property "NEXTSTEP-FUNCTIONS") task) append (list (intern (format nil "~:@(~a~)" property)) (icalsync-cl-coerce-value value))))
Next Steps
There are a few things missing from this implementation. We don't check for the existence of a task before creating it from the corresponding TODO. Priority is not synced. Org-mode is not made aware of items added to Reminders.app. We track where our TODOs are based on the buffer name and their position in the buffer, which will quite easily break if we shuffle our TODOs around. Only the reminder's completion status is synced back to Emacs, but if its description or due date changes, we should synchronize those as well. As a proof-of-concept, however, I think this is a solid start.
In addition, the behavior that you use to turn Org-mode files into lists of reminders – or vice versa – is entirely up to you. There are countless ways to do so. A conversion could, for example, take each calendar, and make it a headline in an Org file, with each task becoming a checklist item with a property drawer. Another conversion could check the TODOs for a custom condition, i.e., only tell iCal about TODOs which have a certain property in their drawer, or are scheduled after a certain date.
All the code written for this article is available in my elstar GitHub repository. The elstar name for my repository is a bannerhead
of sorts, an invitation for this method to be explored further, and to
collect the results in a shared place. I welcome any contributions.
Conclusion
The point of this article was not just to create a synchronization layer for iCal to Org-mode, but to explore the method of synchronization itself. For Emacs users on OS X, the potential exists to fill in the gaps with their usage of operating system (and associated mobile/cloud convenience) and text editor. Some other ideas that may be worth pursuing:
- An Emacs mode for interfacing with iTunes
- Functionality to convert Address Book entries into
BBDB or
org-contactsentries - Turning events on calendars into Emacs diary entries
I look forward to seeing what some enterprising Lisp hackers can come up with.