Notes-mode

Organizing on-line note-taking

Edition $Revision: 1.15 $, for notes-mode version 1.14

$Date: 1998/01/05 22:15:24 $

by John Heidemann

Introduction

What is notes-mode and why should you (perhaps) use it?

What is it?

Notes-mode is an indexing system for on-line note-taking. Notes-mode is composed of two parts, the visible part, a major-mode for emacs to aid note-taking; and the invisible part, scripts which periodically index your notes for you.

Note that notes-mode provides tools to index your notes, not to search them. (Other existing tools such as `grep', `agrep', and `glimpse' already allow file search.)

A digression about indexing vs. searching: Indexing in this sense means organize them according to categories you give, while searching looks through all text for arbitrary strings. Drawing on the World Wide Web for examples, Yahoo (`http://www.yahoo.com/') is an index, while Alta Vista (`http://www.altavista.digital.com/') is a search-engine. In (potentially) more familiar terms, the yellow pages (1) are an index, while directory information (411 in the USA) is sort of a search-engine.

Why keep notes at all?

So why should you use notes-mode? Well, first, consider why you should (perhaps) keep your notes on line. First, I assume that you take notes as part of your work or school. If you don't, you can stop reading now and go back to watching TV.

If you keep notes, ask yourself why you keep them. Reasons vary for different people, but some include:

To remember what is said or done.
To focus on what is important about what is said.
To provide proof of having done something at a particular time or date.
I know there were other reasons here, but they slipped my mind.

Why keep notes on-line?

OK, I've talked you into keeping notes. Why do it on-line? Again, there are different reasons for different people. If you don't want to consider keeping your notes on-line, you're welcome to go back to your (clay tablets) paper notes.

However, if you do much of your work on-line, or if you have portable computer, then you might want to consider keeping your notes on-line.

It's faster to type than write, and possibly more legible at high speed.
Often information is already on-line. For example, in software development, bug reports, measurement results, and everything else that's useful is on-line.
You can take down more detail than you otherwise would (especially if the data is already on-line). Taking more copious notes can be helpful when you go back to figure out why that strange thing was happening.
On-line notes are easy to search. Grep, agrep, and glimpse are all much faster and are often more accurate than paging through paper notes looking for a particular key-word.
On-line notes are easy to index. (At least with notes-mode!) In addition to raw-text search, it's helpful to organize notes by category. If you keep a table-of-contents of your paper notes, you are either extremely fastidious or a librarian (Nadia?).
You can keep all of your notes with you at all times (if you have a portable computer). Even at a page a day, paper notes quickly become bulky and awkward to carry around. On-line notes fit on your computer's hard disk, an extraordinarily compact medium by comparison.
Your notes can be automatically backed up. Paper notes can become damaged with time, and as a graduate student one of my fears was fire in Boelter Hall consuming all my research experiments and and therefore hopes of a degree. Electronic notes are extremely easy to duplicate and can be automatically backed up with the rest of your computer. (You do back up your computer, don't you?)

While these advantages are undoubtedly clear to any right-thinking computer user, it should be said that there are a few disadvantages for on-line note-taking.

If you don't have a computer with you most of time time, it's difficult take notes on-line (because you're off-line, of course). (2)
Computers require power. If your portable computer runs out of juice, you're on your own. Corollary: watch your power, or bring paper. Better corollary: watch your power, and bring paper.
Social limitations. It's not always socially acceptable to take notes-on-line. For example, at a party, few people would use a computer to take down the phone number of a person to whom they're attracted (at least, if they wanted the attraction to be mutual). (3) Sometimes other people find the sound of typing distracting.
Health issues. Repetitive stress injuries do occur writing (slower) by hand is at least an alternate motion than typing.
Legal limitations. If you want to use your electronic notes to justify a patent or invention, you may be breaking legal ground. Being on the legal cutting-edge is rarely an easy thing for the person involved.(4)

Why use notes-mode?

OK, I've sold you on note-taking and even on on-line note-taking. What about notes-mode? Naturally, it slices, dices, and makes julienne fries. But wait, there's more:

It automates indexing your notes, linking notes with the same subject together.
It supports embedded links, allowing you to manually link together different topics and external files.
It includes a number of convenience-features in emacs. Subjects can be completed based on existing subjects. The usual emacs customization mechanisms are available.
Notes containing sensitive information can be encrypted.
Notes-mode seems better than the other, currently available alternatives.

What are the alternatives? I'm glad you asked. (5)

HTML. HTML has better formatting capabilities than notes-mode, and it has excellent linking capabilities. Unfortunately, HTML's tags are fairly intrusive (each is at least four characters long and most come with a pair), tags can get confused with normal text, errors in HTML can be bad (obscuring data), and there's no automatic indexing feature (at least with plain HTML). Besides, all data should be kept as close to the ASCII from whence it came, as God Intended (hi, Steve).
Word Processors. Word processors are strong in the formatting department, but most don't really have linking capabilities, and have poor or restricted indexing.

Y2K Statement

Notes mode uses dates extensively, both two-digit years and seconds-since-1970. However, notes-mode has been coded to function correctly through the year 2038.

To avoid problems with the year 2000, notes-mode assumes that any two-digit years before "70" are 20xx, not 19xx. Notes-mode should therefore work correctly in both the year 1999 and 2000.

Because notes-mode uses seconds-since-1970 for some date calculations it will fail beyond the year 2038 on computers with 32-bit integers.

If I'm still using notes-mode then on a 32-bit machine I'll see what I can do.

Related work

What would a document be without related work?

Notes-mode is not related in any way to Lotus Notes.

I am told (by David Weisman) that it's something like the now defunct Lotus Agenda.

Ashvin Goel, one of the contributors to notes-mode, has gone off and done a from-scratch reimplementation called records-mode. It's very similar to notes mode.

Hyperbole (by Bob Weiner) offers better linking facilities than notes-mode, but it has a bunch of stuff notes-mode doesn't need and it's missing notes-specific indexing provided by notes-mode. For people already using Hyperbole it would be interesting to replace notes-mode's linking with Hyperbole's. Contributions in this area are welcome, provided they make Hyperbole optional.

Basics

All you need to use notes-mode in a chapter. (Except for installation, See section Installation.)

Getting started

To get started with notes-mode, read the introduction this chapter, then run the program `notesinit' to set up everything notes-mode needs. This program will modify your environment (as described in this section), or it will give you the exact commands you should run yourself.

After you've run `notesinit', start up emacs and note-away. I usually begin a day of note-taking by running the command M-x notes-index-todays-link to jump directly to today's note. You may even wish to bind this to something, perhaps with (define-key global-map "\C-cn" 'notes-index-todays-link) in your `.emacs'.

If you want to browse your existing notes, you might instead want to edit the `~/NOTES/index'. (What is a notes file and the index? Hurry up and finish reading this chapter.)

A notes file

The notes file is the focus of most of the activity in notes-mode, it's where you take your notes. Notes files are mostly free-form text broken up into entries. Here's an example:

8-Jun-95 Thursday
-----------------

* Today
-------
prev: <none>
next: <file://~/NOTES/199506/950609#* Today>

next week - release notes-mode

* Environment/notes
-------------------

I explained notes mode to Ashvin and Geoff.
...

Each entry has a subject-block, (maybe) some links, and then (maybe) some text.

The subject-block must begin with an asterisk-space (* ) at the beginning of a line, followed by the subject itself. Subjects must be underlined with a row of dashes (if they're not exact, that's OK; notes-mode will fix them periodically). For convenience, notes-mode will automatically add the underlines when you hit RTN (notes-electric-return), and TAB on a partially completed subject will invoke completion based on indexed subjects (notes-complete-subject).

Following the subject may be links. (In the example, the "Today" entry has links, the "Environment/notes" entry doesn't.) These links will be automatically updated by notes-mode when your notes are re-indexed; just leave a blank line when writing the note.

Links are made with pseudo-URLs, sort of like those in the World Wide Web. Any of these URLs can be followed in notes-mode files by clicking S-mouse-2 on the pseudo-URL (notes-w3-follow-link-mouse).

Finally comes the text. Go wild, but just don't include text that looks like a subject. You can embed pseudo-URLs to link notes together manually.

The more anal of you may have noticed that the lines before the first subject are not part of any entry. These lines are front matter. They're not usually used for much, but they can be a good place to label the file.

There are a number of useful conventions that can be adopted to organize your notes. The most common is the "Today" entry. If you keep an entry with the same subject at the beginning of each file, you link all of your notes together. Other conventions are described later (see section Useful conventions).

The notes index

The notes index lists all subjects you've kept notes about, and each date of each note. Impress your friends, show your advisor why you're worth the big peanuts, you'll soon have the biggest index of all.

The index has one line per subject, listing the subject and each day a note was made about that subject. For example:

Bicycle: 950314, 950316
Bicycle/maintenance/books: 951028
Bridge/hands: 951113, 951114, 951116, 951117
Bridge/UCLA: 960222, 960409

Clicking on any of the dates with mouse-2 will take you to that note (notes-index-mouse-follow-link). (You can also move the point over the date and hit RTN if you're musaphobic [notes-index-follow-link].)

The notes index is automatically updated by the program `mkall'. Typically `mkall' is run nightly by `cron'. On most modern versions of Unix, you can add this command to cron by running `crontab -e' and adding the line:

0 4 * * * /usr/local/lib/notes-mode/mkall

(Assuming that your notes programs are installed in /usr/local/lib/notes-mode, the default location.)

The notes directories

The final thing needed to tie basic notes-mode together his how the pieces fit together. Since my graduate work is in file systems, you can bet that directories are involved.

Notes-mode keeps its files in a two-level hierarchy:

~/NOTES
~/NOTES/index
~/NOTES/rawindex
~/NOTES/199603
~/NOTES/199603/960329
~/NOTES/199603/960330
~/NOTES/199604
~/NOTES/199604/960401

The top level, `~/NOTES', is the notes directory. It keeps all notes in one place. (The name of this directory is configurable, See section Notes-mode configuration.)

Inside the notes directory are two files and a number of directories. The files are `index', the index of all entries (see section The notes index), and `rawindex', used internally.

The notes directory also contains a number of subdirectories, sometimes called intermediate directories. These directories group the actual notes files into manageable chunks, keeping any directory from getting too large. Intermediate directories are named by the four-digit year and the two-digit month of the entries they contain. (The format of intermediate directories is configurable, See section Notes-mode configuration.)

Finally, each intermediate directory are the notes files themselves, named according to the two-digit year, month, and day-of-month.

For the most part, notes-mode will automatically maintain this organization of files, once you create the top-level directory. Notes-mode will also automatically insure that all files in the notes directory are unreadable by anyone other than their owner. Notes are personal things. (This behavior is not currently configurable, but it probably should be.)

Advanced Features

Notes-mode, the minutiae, and some other good stuff.

Notes files

Getting around

Moving between notes entries and around the hierarchy is fairly common, so there are some accelerators.

C-c C-i: Jump to the index entry for the current entry's subject (notes-goto-index-entry).
C-c C-n
C-c C-p: Move to the next or prior note with the same subject (notes-follow-next-link and notes-follow-prev-link). These functions follow the links in the note, if they're defined. If not, they look through the index file. This approach usually works, but will fail if there are multiple new entries created with the given subject between when the index is recomputed.
C-cRTN: Follow the link under the point (notes-w3-follow-link), a keyboard equivalent of S-mouse-2.
M-C-a
M-C-e: Jump to the beginning or end of the current note entry (notes-beginning-of-defun and notes-end-of-defun).
C-c C-k: Copies the pseudo-URL for the current note into the kill-ring (current-url-as-kill). To link two entries, go to the target, grab its URL with C-c C-k, go to where you want to make the link, and yank the URL with C-y.

Notes mode supports imenu, if you have it bound to something (I use (global-set-key [down-mouse-3] 'imenu)).

Subject summary

It's often helpful to look at all entries for a given subject C-c C-s collects all entries with the subject of the current entry in a new buffer (notes-summarize-subject).

Encryption

Notes occasionally contain private material. While Unix has strong services for file protection (compared to other, say, more wide-selling operating systems), in many systems root passwords are shared, while other systems are vulnerable to physical compromise. In such systems, properly used encryption is the best approach to security.

Notes-mode encryption is based Phill Zimmerman's PGP (Pretty Good Privacy) (see `http://www.mantis.co.uk/pgp/pgp.html') and either with Rick Campbell's emacs interface, PAM (PGP Augmented Messaging) (from `ftp://h.gp.cs.cmu.edu/usr/rfb/pam/') (note that as of January 1997, PAM is no longer at this ftp site and appears to not be publically available), or LoPresti and Choi's mailcrypt (from `http://cag-www.lcs.mit.edu/mailcrypt/').

C-c C-e: Encrypt the current note (notes-encrypt-note). By default this function encrypts the whole entry. With a prefix argument, only the part from the point to the end of the entry is encrypted.
C-c C-d: Decrypt the current note (notes-decrypt-note).

By default notes-mode determines your public key by looking up your user-full-name in your PGP keyring. You can override this default by setting notes-encryption-key-id to the desired key-id.

Useful conventions

There are a number of conventions which can make notes-mode easier to use. These conventions are a matter of personal taste, of course. Do what works for you.

First, I find it helpful to keep the date of each notes-file at the top of the file. This makes the file self-identifying if the filename is lost.

Second, I find it useful to have the first entry of each file have the same subject (perhaps "Today"). This entry then links all notes together, making it easy to go to yesterday and tomorrow. I keep a to-do list on this entry, bringing the list forward each day.

A third useful convention is to keep an entry with the name based on the day of the week in each file. Analogous to "Today", this entry links together weeks.

Notes-mode supports these conventions. When you make a new notes-file in emacs, notes-mode searches for the preceding file. If it follows any of these conventions, the new file is initialized appropriately. Currently the approach to do this process (in the program `mknew') is fairly sensitive, so it may not work in all cases. In particular, the date convention works only on for English-language dates. (If you use notes-mode with a non-English language, let me know and I'll work with you to fix this limitation.)

If you find other helpful conventions, please let me know. Modifications to `mknew' to implement new conventions are also invited.

Notes indices

Only two features of notes index mode haven't yet been described. First, you can open any notes-file based on date with notes-index-link, normally bound to o.

Second, you can get a subject-summary with C-c C-s (see section Subject summary). The subject defaults to that of the current index line.

Notes-mode configuration

Several aspects of notes mode are particularly visible to the user. Because I'm not a fascist, a user can change most of these.

Preferences are specified in `~/.notesrc'. This file lists things to change:

# lines beginning with a hash are comments
dir: ~/NOTES
int_form: %Y%m

Currently, two things can be changed:

dir: Specifies the root of the notes directory hierarchy (see section The notes directories).
int_form: Specifies the form of the intermediate directory. A limited subset of strftime(3) formatting is allowed.

The subset of strftime(3) supported in int_form is:

%Y: The four-digit year.
%y: The two-digit year.
%m: A two-digit numeric month.
%d: A two-digit day.

In addition to `.notesrc', there are a number of emacs-specific variables. These variables are documented in the file `notes-variables.el'.

History

More about notes-mode than you wanted to know, and some thanks.

Notes-mode history

Briefly, I started keeping notes on-line shortly after I got a portable computer in January, 1994. After a month-and-a-half of notes, I realized that one does not live by grep alone, so I started adding indexing facilities.

In June of 1995 some other Ficus-project members started keeping and indexing on-line notes using other home-grown systems. After some discussion, we generalized my notes-mode work and they started using it.

Over the next 18 months notes-mode grew. Finally, in April, 1996 I wrote documentation, guaranteeing that innovation on notes-mode will now cease or the documentation will become out of date.

Credits

I (John Heidemann, <johnh@isi.edu>) started, documented, and currently maintain notes-mode. I take ultimate responsibility for the code, especially for the ugly parts that I won't let others change.

Ashvin Goel <ashvin@ficus.cs.ucla.edu> has been a very enthusiastic notes-mode user and contributor. He is responsible for at least the ideas behind notes-summarize-subject and the ideas and initial implementations of some of the original generalization and modularity improvements, notes-follow-next-link and notes-follow-prev-link, notes-goto-index-entry, programmed subject completion, and context-sensitive mouse-2 handling. In addition, he is an invaluable second opinion about what and how things should be done (even if I don't always agree with him).

Geoff Kuenning <geoff@ficus.cs.ucla.edu> has been another enthusiastic notes-mode user and victim. He is responsible for finding several bugs, motivation for mouse-less operation, comments about the documentation, the day-of-week convention, and an initial implementation and the idea of multiple entries with the same subjects in a single notes-file.

Since it's release on Usenet in April 1996 several other folks have contributed. Thanks to David Weisman <weisman@app1.osf.org>, Martin L. Smith <martin@ner.com>, Jason Bastek <jason@aai.com>. See the next section (See section Changes) for details of their exploits.

Changes

For the bored:

First semi-public release. 12-Jul-95: version 0.1 Shared a version with Ashvin and Geoff.

Changed 6-Dec-95: version 0.3 Ashvin's changes for note traversal added (C-c C-p and C-c C-n now move to the prev/next note in note-mode). URL parsing changed so that "localhost" is optional.

Changed 19-Dec-95: version 0.4 More robust prev/next code added, both to handle going back and forward in the middle of chains through the index file, and to handle back/forward in a single file. URL parsing changed so that notes-goto-index-entry correctly handles lookups on notes names such as "252A".

Changed 20-Dec-95: version 0.5. Fixed a missing variable in notes-url.el. Added a work-around to a bug in emacs-19.30's define-derived-mode.

Changed 24-Dec-95: version 0.6. Prev/next code re-re-written to be more robust.

Changed 26-Dec-96: version 0.7. Bug fix release.

Changed 23-Jan-96: version 0.8. Initialization code added to set up a new note. New variable: notes-bin-dir.

I'm skipping version 0.9 because I erroneously release version 0.1 as version 0.9 (only on the web, not on Usenet).

I'm bumping from version "0" to version "1" since the code is has been in production use for more than a year by several people. Minor numbers are the same.

Changed 26-Mar-96: version 1.10. Setup code completely re-written. Several incompatible changes have been made: - the lisp and Perl code must be installed via make install, not by copying. - some data is specified in a .notesrc file; copy and modify sample.notesrc. - several internal elisp changes. - catsubject added (bound to C-cC-s): collect all notes about the current subject. - new notes-files are initialized with fields based on the prior day's notes; see mknew for details. - daily_work is gone; mkall is rewritten to use .notesrc.

Changed 29-Apr-96: version 1.11. Real documentation. Mknew caching added.

Changed 9-Aug-96: version 1.12. Added notesinit to do all setup for new users.

Changed 24-Aug-96: version 1.13. Minor documentation fixes.

Changed 20-Dec-97: version 1.14. Autoconf support.

Fontification of the index buffer is now pre-computed in perl other than done when the file is needed (in elisp with slow regular expressions). 2000-line index files are now 1-2 seconds rather than 15-30 on a 100MHz Pentium. If necessary (the pre-computed version isn't up-to-date) we fall back on the slower code.

Related work improved (suggestion by David Weisman <weisman@app1.osf.org>).

Documentation improvement (problem found by Martin L. Smith <martin@ner.com>).

Installation improved (code by Jason Bastek <jason@aai.com>).

Bug in notes-index mode with subjects containing colons fixed (johnh).

Encryption now supports mailcrypt.el.

Support for emacs 20 (a small font-lock change).

Changed 5-Jan-98: version 1.15. Bug in decryption for non-PAM users fixed (suggestion by Kevin Davidson <tkld@quadstone.com>).

Y2K statement added (suggestion by Kevin Davidson <tkld@quadstone.com>).

Pointer to mailcrypt added (as a supported encryption package). Problem pointed out by K. Ueda <kueda@jupiter.qse.tohoku.ac.jp>.