One day, while I was going through the videos in our archive at the Computer History Museum (CHM), I came across an interesting title: Mockingbird—A Composer’s Amanuensis. I had no idea what “amanuensis” meant (I found out it’s a sort of literary or artist’s assistant), but attached to the tape was a note with names I certainly knew well, Severo Ornstein and John Maxwell. I instantly realized this was the video that I’d been waiting for ever since I started exploring computer music.
Composer John Cage (left) and Lejaren Hiller working on HPSCHD at University of Illinois Experimental Music Center (EMS). Photo: EMS.
Computer music has been an intense research focus since the 1950s. Many institutions, such as the University of Illinois’ Experimental Music Studios and the Columbia-Princeton Electronic Music Centers, saw the potential for computers in music. American composers Lejaren Hiller and John Cage garnered further attention to computer-generated music with their avant-garde creations. As known even to the Ancient Greeks 2,500 years ago, music creation can be modeled as a mathematical process, so designing computer systems to create music was a reasonable possibility. While some composers, like Cage and Hiller, did use computers in the 1950 and ’60s, their use was limited, almost always assisted by nearby technologists. There was no input method that mimicked the natural techniques used by traditional composers and interacting with computers at time proved difficult. It would take several significant breakthroughs in interactive technologies to make the computer a true tool for composers. One of the first attempts at creating such a system was Mockingbird, developed by Ornstein and Maxwell at the Xerox Palo Alto Research Center (PARC).
Screenshot of Smalltalk running on on the Xerox Dorado. Courtesy of the PARC Library.
With some of computing’s boldest thinkers in one place, PARC was the ideal home for a project like Mockingbird. PARC dove headlong into researching new human-computer interaction techniques, ultimately developing many of the technologies that would come to define the personal computer, such as the mouse, the graphical-user interface (GUI), bit-mapped graphics, local area networking, and What You See Is What You Get (WYSIWYG) word processing. The 1972 Xerox Alto computer system, which inspired both Apple and Microsoft to create their own graphical interfaces, originated and incorporated such innovative ideas and established the model for computing we still use today. Building upon the Alto’s intuitive features, Ornstein developed the Dorado system (also known as the Xerox 1132) in the late 1970s, a faster and more powerful version of the Alto.
Ornstein’s early interest in music began with his father, Leo Ornstein, who was a widely admired pianist and composer. (Severo is a strong pianist himself.) He later combined his musical upbringing with technology while analyzing sound with MIT’s large-scale yet “personal” (single-user) TX-2 computer mainframe system in the late 1950s. Realizing that current technology was unable to process audio fast enough, Ornstein put his music explorations on hold and shifted his focus to other areas, such as the development of the LINC (Laboratory Instrument Computer), and the fledgling ARPANET. When Ornstein joined Xerox PARC in 1976, he was instrumental in creating the interface to the world’s first laser printer, used with the Alto system, before being asked to lead the team to develop the Dorado. The suite of programs for the Dorado, as well as its WYSIWYG capabilities, inspired Ornstein to revisit his interest in music by computer.
PARC attracted researchers from around the country to work on various projects in human-computer interaction, as well as networking. In 1978, MIT student John Maxwell III joined PARC as an intern, initially as a part of researcher Alan Kay’s Smalltalk team. Always interested in music, during his second year, Maxwell expressed an interest in writing his PhD thesis around music and the computer. “When I started MIT in 1976, I hoped to find some software for notating music so I could write my music down, but I didn’t find any,” Maxwell recalled. “I was more hopeful about finding software for notating music when I went to PARC as an intern in 1978, but in spite of all of the amazing things going on at PARC, there wasn’t any music notation software. In 1979, Will Crowther, my intern advisor, sent a message out to CSL [Computer Science Laboratory] in PARC asking them what they would work on if they had six months to do anything that they want. I replied that I would like to write a music notation editor. He summarized the responses and sent them out. I was sitting on the floor in Will’s office when Severo came in, all excited, and asked who the person was who wanted to write a music notation editor. Will pointed to me, and Severo looked down at me and said, ‘Oh,’ since I was just an intern. But Will said, ‘No, no! He is a good programmer.’” Like Ornstein, Maxwell had been interested in music from an early age, especially the keyboard.
Musical notation on a Xerox Alto screen
Composers traditionally use staff paper, sheets with five-lined sections (called “staves”) allowing musical notation to be drawn upon them, often while sitting at a piano to play the notes represented on the staff. Earlier systems developed for musical composition did not use traditional musical keyboards, or even computer alphanumeric keyboards, but complicated systems of dials, switches, and patchboards, techniques not easily adapted to use by non-technical users. If Mockingbird was to be of use to a composer with little or no technological experience, it would need to incorporate an easily recognizable entry system. In the case of Mockingbird, this was a piano-style synthesizer.
To incorporate a piano keyboard into Mockingbird, Ornstein and Maxwell devised a system that connected a Yamaha CP-30 synthesizer to the system. This allowed composers to “play in” music directly through a regular piano keyboard. The synthesizer could also play back the music, bypassing the slow and resource-intensive process of sound waveform generation, which could tie up systems. A composer could play the score directly into the computer in a method that felt natural, and then have it played back on the instrument itself.
Once a score is played into the computer, the notes appear on the screen in staves. The composer then works with the score using the Dorado’s mouse to make changes. Mockingbird could even help identify irregularities in the score automatically. Sections could be highlighted, altered, copied, or moved. This work required the confluence of many different technologies. Mockingbird was written in Mesa, a programming language first developed for the Alto. The Cedar graphics program was used to generate the page images of the finished score. By enabling entry via synthesizer, Mockingbird eliminated hand transcription.
In addition to capturing and manipulating scores, Mockingbird connected to a laser printer. While mainframe computers controlling large phototypesetting systems had been used to mass-produce printed scores, the Dorado’s ability to print via the laser printer eliminated the slow and expensive process of phototypesetting. This technique could have made Mockingbird a game changer as far as score publishing went, but the program saw limited use by actual composers.
The opening page of Leo Ornstein’s Seventh Sonata for Piano.
“The closest it got to serious use,” noted Ornstein, “was while my father was writing his seventh piano sonata. He himself was technophobic, so as he finished a few pages of manuscript, he sent them to me. I’d enter them into Mockingbird and send printed copy back to him for correcting. He’d then send back the corrected pages, and I’d put the corrections into the evolving score. That way we slowly assembled a complete score in Mockingbird. Meanwhile as the score grew, I sent printed pages on to the pianist, who was preparing to premier the work in a concert of the San Francisco Contemporary Music Players. It was a roundabout process; but it worked and the score is on the web today here.”
January 1984 issue of Byte Magazine
“Mockingbird made a big splash in the computer music world.” Maxwell said in an email. “I presented Mockingbird to the International Computer Music Conference in Dallas in 1981. It was the last presentation of the conference, but it was still well attended. People in the audience gasped when they saw Mockingbird instantly redraw music after an edit. I also presented Mockingbird at a computer music conference in Zurich, Switzerland. We wrote about Mockingbird for Byte magazine, and it appeared on the cover of the January 1984 issue (along with other things).” I watched the video of Maxwell and Ornstein demonstrating Mockingbird the very morning I discovered it, astounded by what I saw. In the video, Maxwell and Ornstein demonstrate the system in great detail, from entry using the synthesizer through manipulation via the mouse, and playback.
For the last year, I’ve experimented with composition using online music notation systems. As I watched Maxwell interact with the score on the Dorado’s screen, I was struck by how similar the process demonstrated nearly 40 years ago was to the one I used every day. After reaching out to Ornstein and Maxwell for their comments, I discovered that Mockingbird was far more influential than widely used.
Mockingbird—A Musician’s Amanuensis, by Severo Ornstein & John Maxwell, ca. 1980. Collection of the Computer History Museum, 102695167...
“As for Mockingbird,” Ornstein noted in a recent interview, “I think it had most impact as a demonstration of possibilities to show the way for others to develop the follow-on and present-day music managing systems. It never functioned except within the hallowed halls of Xerox PARC on a Dorado with a special logic board and with a specially doctored synthesizer. Several people told me that Mockingbird had had a big impact on the design of their music notation software.”
Maxwell said, “I also heard that a Stanford design course used the Mockingbird video as a model of good design. We considered rewriting Mockingbird for PCs to make it available for general use, but I was more interested in working at PARC and we thought that we had succeeded in getting our ideas out, and it was just a matter of time until good music notation software would show up. However, Severo used Mockingbird to notate much of his father’s music. And I got my 10–20 pieces of music finally written down.”
By the mid-1980s, musical instrument digital interface (MIDI) revolutionized the use of computers in music. The rise of personal computers brought with it many software packages incorporating features Mockingbird had originated. Programs, such as SynthLAB for the Apple II, ENCORE for the Atari ST series, and the multiplatform Music Construction Set brought composition capabilities to the masses, while high-end systems from companies like Fairlight and New England Digital produced systems that integrated the keyboard into the creation and composition process.
Now, dozens of software programs allow even amateurs like myself to create music, and several online composition sites allow original music to be composed, performed, and shared. Today, nearly all composers use computers for writing scores and with the ease computer composition imparts to the process, many hobbyists have come to composing their own works using both personal computer-based and web-based programs. Mockingbird was the heroic opening movement in a now long symphony of music composition software.
