Can we use machine learning to create compelling art and music? If so, how? If not, why not?
Graphical (drawn) Sound is a technology for synthesizing sound from light that was invented in Soviet Russia in 1929 as a consequence of the newly invented sound-on-film technology. Pavel Tager initiated developments of the first Russian sound-on-film systems in 1926 in Moscow and in 1927, just a few months later, Alexander Shorin started his research in Leningrad. Tager’s system, the Tagephon, was based on intensive variable density optical recording on film while in Shorin’s Kinap system the method of transversal variable area optical recording on film was realized. Another version of Shorin’s system, the Shorinophone, which was widely used for field and studio sound recording, was based on the mechanical reproduction of gramophone-like longitudinal grooves along the filmstrip.
Among the first Soviet sound movies ever created was "The 5-year Plan" (Piatiletka. The Plan of Great Works) by Abram Room. The group working on this film in 1929 at Shorin’s Central Laboratory of Wire Communication in Leningrad included the painter, book illustrator and animator Mikhail Tsekhanovsky, the chief of the composer’s brigade Arseny Avraamov and the inventor Evgeny Sholpo. Meanwhile the sound-on-film techology made possible access to the sound as a trace in a form that could be studied and manipulated. It also opened up the way for a systematic analysis of these traces such that they could be used to produce any sound at will. When in October of that year the first roll of film was developed, it was Tsekhanovsky who voiced the idea: “What if we take some Egyptian or ancient Greek ornaments as a sound track? Perhaps we will hear some unknown archaic music?”  He was referring to the shapes and outlines of vases and how these could be used like wave forms to generate sound. It was at this precise moment that graphical sound techniques were invented. The next day they were already furiously at work on experiments in what they referred to variously as ornamental, drawn, paper, graphical, artificial or synthetic sound. The laboratories that were soon created became the first-ever prototypes of the future centres for computer music. While most inventors of electronic musical instruments were developing tools for performers, the majority of methods and instruments based on Graphical Sound techniques were created for composers. Similar to modern computer music techniques, the composer could produce the final synthesized soundtrack without need for any performers or intermediates.
In December 1930 Mikhail Tsekhanovsky wrote in his article About the Drawn Sound Film: "with the invention of new drawn sound techniques (developed by Arseny Avraamov in Moscow, Sholpo and [Georgy] Rimsky-Korsakov in Leningrad) we are achieving a real possibility of gaining a new level of perfection: both sound and the visual canvas will be developing completely in parallel from the first to the last frame […] Thus the drawn sound film is a new artistic trend in which for the first time in our history music and art meet each other." 
At exactly the same time similar efforts were being undertaken in Germany by Rudolf Pfenninger in Munich and, somewhat later, by Oscar Fischinger in Berlin. Serious research was conducted in Moscow by Boris Yankovsky. Because of the cross-disciplinary nature of the new technique, people involved in it had to be skilled not only in music, but in acoustics, mathematics, sound-on-film technology and engineering. As a result even skilled journalists often could not understand the physical meaning of the phenomena under consideration or specific technological ideas. Having no developed terminology, many mistakes and unexpected “puzzles” appeared in their writings. Moreover, there were several known research groups – competitors in Russia and Germany working in parallel. It led to a very specific problem – encryption of the information. For instance, in the well-known photograph Oscar Fischinger holds ‘fake' rolls made by his Studio for publicity purposes as he did not want his competitors to learn his actual techniques. He never used rolls as large as this – they were fakes.  Yankovsky had a very special way of making notes on his ideas. It is impossible to understand the construction of his tools from reading one description without referring to several other manuscripts that offer important keys for understanding it.
Although there were several short articles published in German, French and English [4-6] most publications about research and developments in the USSR were only in Russian. At the same time most important documents were never published at all and were circulating only in manuscript form, similar to ‘Samizdat’ (self-published forbidden literature).
By 1936 there were several main, relatively comparable trends of Graphical Sound:
- Hand-drawn Ornamental Sound, achieved by means of shooting still images of drawn sound waves on an animation stand, with final soundtracks produced in a transversal form (Arseny Avraamov, early Boris Yankovsky);
- Hand-made Paper Sound with final transversal soundtracks (Nikolai Voinov);
- The Variophone or Automated Paper Sound with soundtracks in both transversal and intensive form (Evgeny Sholpo, Georgy Rimsky-Korsakov);
- The Syntones method, based on the idea of spectral analysis, decomposition and resynthesis, developed in 1932-1935 by a pupil of Arseny Avraamov, the young painter and acoustician Boris Yankovsky.
Among the researchers working with Graphical Sound after World War II were the famous filmmaker Norman McLaren (Canada) and the composer and inventor Daphne Oram (UK).
The ornamental sound technique, developed in 1929-1930 by Arseny Avraamov, was similar to German animator and filmmaker Oscar Fischinger's sounding ornaments first presented in 1932. In 1930, however, Avraamov was the first to demonstrate experimental sound pieces – based on geometric profiles and ornaments – produced purely through drawing methods. This was achieved by means of shooting still images of drawn sound waves on an animation stand.
In autumn 1930 Avraamov founded the Multzvuk group at Mosfilm Productions Company in Moscow. To produce his first drawn ornamental sound tracks he had on staff a special draughtsman, cameraman Nikolai Zhelynsky, animator Nikolai Voinov and acoustician Boris Yankovsky who was responsible for the translation of musical scores into Avraamov's microtonal Welttonsystem as well as Samoilov's Ober-Unter-Tone Harmony system. The final scores were coded in Yankovsky's 72- step ultrachromatic scale with the dynamics and speed variations indicated by the number of frames. Yankovsky was also involved in the production of acoustic experimental studies, developing methods for the synthesis of sounds with glissando, timbre crossfades, timbre variations and polyphony by means of multiple shooting on the same optical soundtrack (alternative to multi-track recording which was not available yet).
From 1930-34 more than 2000 meters of sound track were produced by Avraamov's Multzvuk group, including the experimental films Ornamental Animation, Marusia Otravilas, Chinese Tune, Organ Cords, Untertonikum, Prelude, Piruet, Staccato Studies, Dancing Etude and Flute Study. In autumn 1931 the Multzvuk group moved to NIKFI (Scientific Research Institute for Cinema and Photography) and was renamed Syntonfilm Laboratory. In December 1932 NIKFI stopped supporting Syntonfilm and the group moved to Mezhrabpomfilm where in 1934 it was closed as it was unable to justify itself economically. The whole archive was kept for many years at Avraamov's apartment, where in 1936-37, during Avraamov's trip to the Caucasus, it was burned by his own sons, making rockets and smoke screens with the old nitro-film tapes, which were highly flammable.
In 1931 Nikolai Voinov left Multzvuk group and started his own research as a developer of paper sound techniques. These were based on the synthesis of sound waves by means of paper cutouts with the carefully calculated sizes and shapes produced by his newly invented tool, the Nivotone. Voinov's method offers a surprisingly efficient level of control over the dynamics of sound.
As of 1931 Voinov was involved in the activities of the IVVOS group (Ivanov, Voinov, Sazonov). This group produced a number of animated cartoons with synthetic sound tracks, including Barynia (1931), Rachmaninov's Prelude (1931), The Dance of the Crow (1933), About a Pig's Snout and a Soviet Kitchen Garden (1933),The Thief (1934) and Zones, Safety Lines (1934). In early 1936 Voinov was dismissed from the Moscow Film Factory and his laboratory was closed. For the rest of his life he worked successfully as a cameraman at Souzmultfilm Studio. In Voinov's official biography his most experimental work from 1931-36 is not even mentioned.
The Variophone was invented by Evgeny Sholpo in 1930 at Alexander Shorin's Central Laboratory of Wire Communication in Leningrad. In May 1930 Sholpo applied for a patent on a “method and device for the production of a periodic sound track on film.” It was a continuation of research that Sholpo had been conducting since the 1910s when he was working on performer-less music.
The first version of the instrument was built with assistance from the composer Georgy Rimsky-Korsakov (grandson of the famous composer Nikolai Rimsky-Korsakov) in 1931 at Lenfilm Studios. It was capable of producing artificial soundtracks by means of Automated Paper Sound techniques. Sholpo’s method made access easier to varieties of timbres. Unlike Avraamov, who shot still images of sounds on an animation stand, Sholpo used cardboard disks with circular images of combs with suitably shaped cogs rotating synchronously with a moving filmstrip. The advantages of the Variophone were in its flexible and continuous pitch control and vibrato.
Although the very first version of the Variophone was made with wooden parts fixed by wires and tuned with ropes, it already incorporated one of the most crucial and necessary devices – a mechanism for the precise and continuous changing of the speed of rotation of the optical disk with the sound wave pattern i.e. a means of controlling the pitch with the possibility of synthesizing continuous glissandos. Also from the very beginning the composer had full freedom to work with polyrhythmic combinations and almost unlimited tempos in any passages.
By the early 1931 at Lenfilm Studio, with assistance from G. Rimsky-Korsakov, Sholpo made a soundtrack for the short propaganda film The year 1905 in Bourgeois Satire (director N.I. Galkin, composer V.M. Deshevov). In the summer of 1932 Sholpo and Rimsky-Korsakov produced the synthesized soundtrack for the new color cartoon The Symphony of Peace by E.J. Johansson and G.V. Bankovsky.
Many soundtracks for movies and cartoons were produced using the Variophone. Among the most accomplished pieces recorded with the Variophone in 1933-34 were The Carburettor Suite by G. Rimsky-Korsakov, Waltz by N. Timofeev, Flight of the Valkyries by Richard Wagner, and Franz Liszt's 6th Rhapsody.
The Variophone was continuously under development and by 1936 the arsenal of musical and acoustical means of the second version was enriched highly with possibilities of free glissando with a speed of up to four octaves per second, flexible and exact control over dynamics, and options for deep vibrato for pitch, volume and timbre. The Variophone could produce polyphonic soundtracks with up to twelve parallel voices. Even compared with the subsequent and more advanced third and fourth versions, it produced the most impressive quality and complexity of sound. In 1941 during the blockade of Leningrad, together with composer Igor Boldirev, Sholpo synthesized one of his most experimental pieces – the soundtrack for the cartoon The Vultures. Although aesthetically these works are similar to Walter Carlos' Switched-on Bach (1968) and sound like eight-bit music, the main difference is in their timing. In 1918 Sholpo developed special tools - the Melograph and Autopianograph - to register the temporal characteristics of live musical performance. Much electronic music has a rigid tempo, like a metronome; Sholpo was able to simulate more subtle variations in tempo such as rubato, rallentando and accelerando, based on his careful analyses of live piano performances by the best pianists.
Syntones and Audio Computing
In 1931-32 Boris Yankovsky (1904-1973) was on the staff of the Multzvuk group. In 1932, however, disappointed with its Ornamental Sound approach, he left the group. Unlike most of his colleagues he understood that the waveform does not represent the tone colour uniformly and that only the spectrum of sound developed in time with all the nuances of its temporal transitions can give a complete picture. Of all the early graphical sound pioneers, Yankovsky alone pursued the approach of spectral analysis, decomposition and re-synthesis. His concept was based on the belief that it is possible to develop a universal library of sounds similar to Mendeleev's table of chemical elements. His curves were spectral templates, semiotic entities that could be combined to produce sound hybrids. As an option he developed several sound processing techniques including pitch shifting and time stretching based on the separation of spectral content and formants, resembling recent computer music techniques of cross synthesis and the phase vocoder. To realize these ideas he invented a special instrument, the Vibroexponator – the most paradigm-shifting proposition of the mid-1930s.
In 1935 in one of his manuscripts Yankovsky wrote: ‘It is important now to conquer and increase the smoothness of tone colours, flowing rainbows of spectral colours in sound, instead of monotonous colouring of stationary sounding fixed geometric figures [wave shapes], although the nature of these phenomena is not yet clear. The premises leading to the expansion of these phenomena – life inside the sound spectrum – give us the nature of the musical instruments themselves, but “nature is the best mentor” (Leonardo da Vinci) […] The new technology is moving towards the trends of musical renovation, helping us to define new ways for the Art of Music. This new technology is able to help liberate us from the cacophony of the well-tempered scale and related noises. Its name is Electro-Acoustics and it is the basis for Electro-Music and Graphical Sound’. 
Yankovsky's approach had much in common with that of Rudolf Pfenninger. Both were primarily focused on acoustics. As Thomas Levin put it: ‘ Fischinger's curves are not derived from sound, they generate it, whereas Pfenninger's curves are in the last analysis derived from the sounds that they analytically recreate… Pfenninger's curves are decidedly not ornaments but are rather, as numerous critics have rightly noted, “templates or print-types”, that is, semiotic entities that can be combined to produce sounds in a linguistic – which is to say, thoroughly technical and rule-governed – manner. Unlike Fischinger's curves, which were continuous, Pfenninger's were discrete units'. 
The method developed by Boris Yankovsky was based on pure audio computing techniques and possessed properties, very common for digital technologies, such as discretization and quantization of audio signals and related spectral data, manipulation with ready-made parts, and operations with selections from databases of the basic primitives (templates). To perform complex mathematical calculations of waveforms as well as other important parameters of sound and automated musical performance such as rhythm, there were special ‘employee-computers’ on the staff in the laboratories of Boris Yankovsky and Evgeny Sholpo. These were mathematicians whose specific task it was to make calculations.
In 1933 Yankovsky was invited to Mosfilm Studios to organize the Laboratory for Synthetic Sound Recording, where in 1934-35 he recorded a sizeable collection of samples of instruments from the Symphony Orchestra of the Bolshoi Theatre. By 1936 the collection of 110 synthesized templates was created. In 1935 Yankovsky joined the Autonomous Research Section (ANTES) at the Union of Composers in Moscow, founded by commissar Boris Krasin, Avraamov and Ogolevets. It was the last significant manifestation of creativity with its roots in the forward-looking 1920s. In 1936 the infamous Pravda article Confusion instead of music was published, initiating a war by the totalitarian state against the freedom of artistic expression. After the death of Krasin in 1936 ANTES was closed and the Ministry of Culture stopped funding Yankovsky's laboratory. It was passed to the NIMI institute at Moscow Conservatory. Little changed, but by the end of 1937 he finally got his syntones to make sounds.
In 1939 Yankovsky met Evgeny Murzin – a young inventor, fascinated by the idea of a universal tool for sound synthesis. The same year Boris Yankovsky and Evgeny Sholpo decided to unite their efforts and the new Laboratory for Graphical Sound at the Institute of the Theatre and Film was established. Yankovsky moved to Leningrad. He expected to complete the final version of his vibroexponator in 1940 but finally was prevented by World War II and never returned to the graphical sound.
The ANS Synthesizer
In 1957 the young inventor Evgeny Murzin (1914-1970) finished developing and patented a photo-electronic musical instrument called the ANS Synthesizer. Its name was derived from the initials of influential composer Alexander Nikolayevich Scriabin. It was remarkably close to the concept of Evgeny Sholpo's Mechanical Orchestra. The instrument was based on a set of sine wave oscillators, produced by pure optical means as multiple (144) graphical soundtracks printed on four glass disks, adjusted on fixed frequencies, forming a discrete pitch scale, covering the whole audible range with intervals between successive pitches imperceptible to the human ear. Control over the system and the process of sound synthesis was carried out by means of a special graphical score with a diagram representing the spectrum of a sound by means of drawn transparent strips, having appropriate shapse and slopes, making it possible to operate the whole set of sine wave tones synchronously and independently, controlling the sound on a spectral level, directly manipulating the overtones, erasing the difference between the pitch harmony structures and a spectral tissue of a sound. A similar principle of the graphical score was used later in the legendary UPIC computer system, developed by Yanis Xenakis in 1977 in the Centre d'Etudes de Mathematiques et Automatiques Musicales in Paris.
The instrument was based on the scale of 72 steps per octave proposed by Boris Yankovsky, who was involved in the development of the ANS in 1939-40. The synthesizer is based on the same principles as the variophone. It incorporates a set of rotating optical disks with photo-printed round optical sound tracks. While in the variophone one rotating disk produced a single sound, in the ANS each optical disk contained 144 independent sound tracks. Four disks, used in the first version of the instrument, could produce simultaneously 576 sine waves with frequencies covering the whole audible range. ANS was a real-time instrument, producing the audible result directly during work.
In 1967 the studio of electronic music was established in Moscow, with the ANS synthesizer at its core. Among the composers working with the ANS were Alfred Shnitke, Sofia Gubajdulina, Edison Denisov, Eduard Artemyev, Alexander Nemtin and Stanislav Krejchi. The instrument was used for scoring many films, in particular the early films of Andrei Tarkovsky.
 Avraamov, Ars. “Sinteticheskaya muzika” Sovetskaya Muzika , 1939, No.8, pp. 67-75
 Tsekhanovsky M. 1930. “O Zvukovoi Risovannoi Filme.” Kino I Zhizn, Moscow. 1930, no. 34-35, p.14
 According to information received from the Fischinger Trust, the Center for Visual Music
 Von A. Leo. “Die Grosse Erfindung – ‘Graphomusik’”, Moskauer Rundschau, 15.03.1931 Le “Variophone”, Le Journal de Moscou, 01.06.1935
 Sоlеv, V. “Absolute Music by Designed Sound.” American Cinematographer. April, 1936, pp. 146-148, 154-155
 Yankovsky, B. “Analiz i sintez tembra” (Analysis and Synthesis of Timbre) March, 1935, Moscow. Levin, T. “Tones from out of Nowhere: Rudolf Pfenninger and the Archaeology of Synthetic Sound.” Grey Room 12 (Fall 2003): p. 32-79
Unpublished article. Theremin Centre Archive. p. 35
 Smirnov A., Pchelkina L., 1917-1939. Son Z / Sound in Z. PALAIS / Palais de Tokyo Magazine, Paris, 2008, №7, pp. 66-77 (English and French).
 Smirnov A., Pchelkina L., Les Pionniers Russes de'l ART du SON. Experimentations musicales. Catalogue of the exhibition Lenin Stalin et la Musique at Cite de la musique, Paris, 2010. (French).
 Smirnov A., Son produit par la lumiere et le papier. Catalogue of the exhibition «Vois ce que j'entends», Centre des Arts Enghien-les-Bains, France, 2010. (Russian and French).
 Smirnov A., Boris Yankovsky: Leben im Klangspektrum. Gezeichneter Klang und Klangsynthese in der Sowjetunion der 30er Jahre. Klangmaschinen zwischen Experiment und Medientechnik. Transcript Verlag, Bielefeld. Austria, 2010. (German).
 Smirnov A., Boykett T., Notation und visuelle Musik. Klangmaschinen zwischen Experiment und Medientechnik. Transcript Verlag, Bielefeld. Austria, 2010. (German).
 Смирнов А., Пчелкина Л., ПОКОЛЕНИЕ Z. Booklet of the Generation Z exhibition. Polytechnic Museum, Moscow, 2011. (Russian). Smirnov A., Pchelkina L., GENERATION Z. Booklet of the Generation Z exhibition. OSA Archivum, Budapest, 2011. (English and Hungarian).
 Smirnov A., Pchelkina L., Russian Pioneers of Sound Art in the 1920s. Catalogue of the exhibition 'Red Cavalry: Creation and Power in Soviet Russia between 1917 and 1945'. La Casa Encendida, Madrid, 2011.
(English and Spanish).
I would like to thank Lev Bolotsky and Marina Sholpo for unique documents which they rescued from destruction and kept for so many years. I also thank Nikolay Izvolov for information and advice, Matthew Price and Rob Mullender for their help in preparation of texts.
Andrey Smirnov, Moscow, 2011
|(C) Andrey Smirnov, 2006 :: Updated 9 November, 2011||Tomado de http://asmir.theremin.ru/graphical_sound.htm|