SwingOSC - Hands-On (SuperCollider Symposium 2007)

last mod: 08-aug-09 sciss (This is a slightly updated version of the script from the symposium)

I. Prerequisites

What is SwingOSC?     Swing ... OSC ...

 

Why SwingOSC?

 

Requirements

 

II. First Steps

The first step is launch the SwingOSC server. The plain way to do it, is to

OSC comes in two common "transport flavours" :

 

Using the -u option will use UDP, while -t selects TCP. The <portNum> argument is necessary because there may be different UDP or TCP services per computer, so they are distinguished by their port (an integer number). For example, SuperCollider language uses UDP port 57120, while the audio synthesis server (scsynth) uses UDP port 57110 by default.

For simplicity, we use the ready-made shellscript SwingOSC_TCP.command (Mac OS X) or SwingOSC_TCP.sh (Linux) to use TCP on port 57111. (Sorry the .bat windows script seems to missing at the moment, but you can use the one that comes with Psycollider). This also uses the -L option which forbids access from remote computers, so you will need to remove the -L when your GUI server is supposed to be on a computer different from the one running SuperCollider!

Now let's see if the server responds. We do it in a very low level way, so it becomes more transparent what's happening behind the scenes. In SuperCollider we need to create a TCP client socket connected to the server:

    n = NetAddr( "127.0.0.1", 57111 );
    n.connect;  // necessary for TCP, for UDP omit this line!
    n.sendMsg( '/print', '[', '/local', \userName, '[', '/method', 'java.lang.System', \getProperty, 'user.name', ']', ']');
	

The last line looks a bit complicated, especially if you are not familiar with the java API. Don't worry, the low-level communication will disappear under the hood in a second. Just check that the your user name is correctly printed in the terminal.

Now let's see what we can do with the existing GUI classes of SuperCollider:

    g = SwingOSC.default;
    g.connect; // only necessary when you start SwingOSC without the -h option, so leave it away when using the .command or .sh shell scripts
    JSCWindow.viewPalette;
	

View Palette with Aqua LAF View Palette with Motif LAF

The left window is the default look on Mac OS X. The right window was produced by changing the look-and-feel first, using this line:

    g.sendMsg( '/method', 'javax.swing.UIManager', \setLookAndFeel, "com.sun.java.swing.plaf.motif.MotifLookAndFeel" );
    JSCWindow.viewPalette;
	

Note: to return to aqua look-and-feel (on Mac), use "apple.laf.AquaLookAndFeel".

Some of the standard gadgets are rendered using the look-and-feel, as you see, for example the JSCSlider, the JSCPopUpMenu etc. Others – like JSC2DSlider or JSCButton – have been customly written for SwingOSC and look the same on all platforms and with all look-and-feels. (see also: www.javootoo.com)

Note: the variable g now holds the instance of the default SwingOSC server. The SwingOSC class is modelled after the Server class (the client representation of scsynth). The method sendMsg sends an OSC message to the server.

Note: SuperCollider only knows four types of OSC arguments: Strings (s), Integers (i), Floats (f), and Blobs (b). The mixed use of Strings ("com.java...") and Symbols ('/method', \setLookAndFeel) is a mere question of taste here.

The OSC command here follows the pattern '/method', <objectID or className>, <methodName> ... <methodArgs>
See also the file OSC-Command-Reference.html in the SwingOSC folder.
For the above call, consult also the API documentation: java.sun.com/j2se/1.4.2/docs/api/javax/swing/UIManager.html

 

III. Building our first Window

While you can talk directly (low-level) to GUI java classes with SwingOSC, e.g. instantiate a javax.JFrame and inside a javax.JButton, we are going to use high-level classes in SuperCollider which are more or less closely linked to java counterparts on the server. So at the moment we will forget about the details of the java world.

The starting point for every GUI is a window:

    w = JSCWindow.new;  // this creates the window (it's still invisible)
    w.front;                    // this makes the window actually visible

The window implies a container view (it's the socalled JSCTopView) which can be filled with child components.

Note: I'm going to use the term 'component' synonymously with 'view', sometimes 'gadget' (where 'component' is more general as it can be another container view, and 'gadget' sounds more like it's a button or slider etc.)

Child components can be buttons, sliders, popup-menus, envelope-views etc. Each component is created with a Rect argument to specify its bounds inside the parent view (the window), and gets automatically added to the parent view:

    // creates and adds a 2D-Slider inside the window w:
    x = JSC2DSlider.new( w, Rect( 10, 10, 160, 160 ));

The user can now interact with the GUI, but we need a means to be notified about its actions. Most gadgets allow you to assign an action-function that gets called whenever the user modifies the gadget's state (e.g. drags the slider in the example above):

(
    x.action = { arg view;  // the argument to the action function is the component
        ("The slider's value is now " ++
            view.x.round( 0.01 ) ++ " / " ++
            view.y.round( 0.01 )).postln;
    };
)

There are more specialized action functions that can be assigned: Actions for keyboard typing (keyDownAction and keyUpAction), actions for mouse control (mouseDownAction, mouseUpAction, mouseOverAction, mouseDragAction), actions for handling drag-and-drop (canReceiveDragHandler, beginDragAction, receiveDragHandler), an action when the component is removed (onClose). Here is an example:

    // first create a second slider component
    y = JSC2DSlider.new( w, Rect( 200, 10, 160, 160 ));

// a copy+paste logic: pressing 'c' copies the x and y value, 'v' pastes
// the values (try to copy from the left to the newly created right view!)
(
    var clipboard, func;

    func = { arg view, char, modifiers, unicode, keycode;
        var handled;
        
        ("Pressed char is '" ++ char ++ "'").postln;
        
        switch( char,
        $c, {
            "Copy!".postln;
            clipboard   = view.x @ view.y;
            handled     = true;
        },
        $v, {
            if( clipboard.notNil, {
                "Paste!".postln;
                view.x = clipboard.x;
                view.y = clipboard.y;
            });
            handled = true;
        });
        // if the result of the keyDownAction is not nil,
        // the key press is 'consumed' (not processed by any
        // of the component's parent views)
        handled;
    };
    x.keyDownAction = func;
    y.keyDownAction = func;
)

Another example for mouse control:

(
// Colorize the view's background while dragging the mouse
[ x, Color.red, y, Color.blue ].pairsDo({ arg view, color;
    view.mouseDownAction = { arg view, x, y, modifiers, buttonNumber, clickCount;
        view.background = color;
    };
    view.mouseUpAction = { arg view, x, y, modifiers, buttonNumber, clickCount;
        view.background = Color.clear;
    };
});
)

IV. Using the GUI.* Syntax

If you are not planning to mix your GUI with custom java components, are merely relying on the ready-made component classes that come with SwingOSC, and you are giving away your code to other people, it is highly recommended to make an abstraction from the actual component classes (such as JSCWindow, JSC2DSlider, etc.).

Instead you use a special factory class called GUI. Using this class, your GUI code can be rendered with other GUI libaries, not just SwingOSC. For example, on Mac OS X, you can choose to present the GUI using the original Cocoa GUI classes, and some basic classes already exists for an Emacs integrated GUI.

Using GUI is straightforward: To create a window, instead of JSCWindow.new you write GUI.window.new. To create a 2D-Slider, instead of JSC2DSlider.new, you write GUI.slider2D.new. The names of the components can be looked up the GUI help file. Here is code from above in the abstracted version; we render it twice with SwingOSC and Cocoa GUI-Kits (the latter only works on Mac OS X!):

(
[ \swing, \cocoa ].do({ arg name, i; GUI.useID( name, {
    w = GUI.window.new( name.asString, Rect( 200 + (i * 440), 200, 400, 200 ), false );
    2.do({ arg j; GUI.slider2D.new( w, Rect( 10 + (j * 200), 10, 160, 160 ))});
    w.front;
})});
)

V. Discovering "plusGUI" Methods

SuperCollider comes with a bunch of useful built-in visualizations and GUI-controls. They are accessed by calling special methods on objects that can be visualized (such as an array of numbers) or controlled by a GUI. They use the current GUI kit which can be switched using GUI.swing or GUI.cocoa. For example, every object can be "inspected" (all its fields are shown, those with setters can be modified):

    GUI.swing;  // or GUI.cocoa if you like
    Server.default.options.inspect;

The inspector shows the current field values using a JSCDragSource (for read-only fields) or a JSCDragBoth (for read-and-write fields):

JSCDragBoth Screenshot

You can thus modify the fields with simple drag-and-dropping. Here is a window to select a sampling rate from:

(
    var rates = [ 44100, 48000, 88200, 96000 ], dragSource;
    w = GUI.window.new( "SR", Rect( 600, 300, 128, 72 ), false );
    dragSource = GUI.dragSource.new( w, Rect( 4, 34, 120, 26 ))
        .object_( rates.first );
    GUI.popUpMenu.new( w, Rect( 4, 4, 120, 26 ))
        .canFocus_( false ) // disable ugly focus border, we don't need it
        .items_( rates.collect( _.asString ))
        .action_({ arg view; dragSource.object = rates[ view.value ]});
    w.front;
)

Another useful "plusGUI" is browse which can be called on any class:

    JSCDragView.browse; // show the class browser for JSCDragView

To visualize data, plot and scope can be used. plot works "offline" and can be called on an Array, Signal, Buffer or Env object. scope is a realtime tool and can be called on a UGen-Graph-Function, a Server or a Bus:

    // 1000 samples from the cauchy distribution centered around 0.0
    Array.fill( 1000, { 0.cauchy }).plot;
    // a basic envelope
    Env.linen( attackTime: 0.1 ).plot;
    // microphone input signal
    s.waitForBoot({ Bus( \audio, s.options.numOutputBusChannels, 1 ).scope })
    // some synth
    s.waitForBoot({{ Saw.ar( mul: 0.25 )}.scope })

 

VI. Building User Views with JPen

Sometimes the ready-made components that come with SwingOSC are not sufficient for your GUI demands. In this case, you have two options: either you develop a custom Java (Swing) component – something we will be looking at in chapter IX –, or (a bit easier) you develop a custom component in SuperCollider, using the JSCUserView class. A JSCUserView at first is a very plain thing. The actual component rendering is performed by assigning a drawFunc function which utilizes the special JPen class. JPen contains methods for painting basic shapes such as lines, rectangles, circles etc. Here is a simple peak meter view:

(
    // we store the current GUI and it's pen class (e.g. JPen)
    // in a variable because they might change while the component
    // exists and would thus produce an error when the Swing
    // user view tries to render using the cocoa Pen...
    var gui = GUI.current, pen = gui.pen, pp = 0,
        numSegments = 8, decibelsPerSegment = 4.5, colors,
        synth, resp;

    colors = Array.fill( numSegments, { arg i;
        Color.hsv( i / numSegments * 0.5, 1.0, 0.5 );
    });
    
    w = gui.window.new( "Meter", Rect( 200, 200, 128, 200 ));
    w.view.background = Color.black;
    v = gui.userView.new( w, Rect( 44, 4, 40, 192 ))
        .canFocus_( false )  // so we don't see the focus border
        .resize_( 4 )  // the view grows vertically when the window is resized!
        .drawFunc_({ arg view; var bounds, peakSeg;
            // view.bounds returns the rectangle bounds of the view
            // relative to the top left corner of its window
            bounds = view.bounds;
            // to simplify drawing we shift and scale the coordinate system
            pen.translate( bounds.left, bounds.top );
            pen.scale( bounds.width, bounds.height );
            peakSeg = (pp.ampdb.neg / decibelsPerSegment).clip( 0, numSegments ).asInteger;
            if( peakSeg < numSegments, {
                (peakSeg .. (numSegments-1)).do({ arg i;
                    pen.fillColor = colors[ i ];
                    pen.fillRect( Rect( 0, i / numSegments, 1, 0.8 / numSegments ));
                });
            });
        });

    s.waitForBoot({
        synth = { var inp, peakPeak, trig;
            inp = AudioIn.ar( 1 );
            trig = Impulse.kr( 20 );
            peakPeak = RunningMax.ar( inp, trig ) - RunningMin.ar( inp, trig );
            SendTrig.kr( trig, 0, peakPeak );
        }.play;
        resp = OSCpathResponder( s.addr, [ '/tr', synth.nodeID ], { arg time, resp, msg;
            pp = msg[ 3 ];
            { v.refresh }.defer;
        }).add;
    });
            
    // a function that get's called when the window is closed:
    // stop the metering synthesizer
    w.onClose = { synth.free; resp.remove };
    w.front;
)
    

Note: the view is repainted using v.refresh. This is placed inside a { }.defer block in order to make it compatible with cocoa GUI. While swing GUI doesn't have that restriction, in cocoa GUI (Mac OS X native) methods on components can only be called inside the AppClock thread. { }.defer makes sure its body is executed on that thread.

 

VII. Model-Controller-View

When designing a GUI, there is a useful pattern that we can follow. It is called MCV = Model-Controller-View because it divides the interactivity process into these three parts:

MCV Diagram(public domain via en.wikipedia.org)

The idea is that we have some object that can be manipulated, the model. The model is visually presented by the view and manipulated by the view or any other controller (such as evaluating text in SC, or MIDI input etc.). The crucial point is that the model doesn't know about the view, hence the user interface can be changed or omitted later without destroying the code or loosing functionality.

My suggested way of implementing a MCV like structure in SC is to use a very basic mechanism that is built into every Object: Dependant-registration. It works like this:

    ~model          = Dictionary.new;
    ~ctrlSet        = { arg key, value; ~model.put( key, value ); ~model.changed( key, value )};
    w               = GUI.window.new.front;
    ~viewA          = GUI.slider.new( w, Rect( 4, 4, 380, 26 ));
    ~viewB          = GUI.slider.new( w, Rect( 4, 34, 380, 26 ));
    ~ctrlA1         = { arg view; ~ctrlSet.value( \a, view.value )};
    ~viewA.action   = ~ctrlA1;
    ~viewA.onClose  = { ~ctrlA2.remove };
    ~ctrlA2         = Updater( ~model, { arg obj, key, val; if( key === \a, {{ ~viewA.value = val }.defer })});
    ~ctrlB1         = { arg view; ~ctrlSet.value( \b, view.value )};
    ~viewB.action   = ~ctrlB1;
    ~viewB.onClose  = { ~ctrlB2.remove };
    ~ctrlB2         = Updater( ~model, { arg obj, key, val; if( key === \b, {{ ~viewB.value = val }.defer })});
    

The Updater class calls addDependant on the model. The model keeps a list of dependants. When the model's changed method is called, all dependants are notified about the change and can act accordingly. This way we can add logic that operates on the model without having to know about all the dependants (i.e. view or the controller for the view):

    ~rout = fork { inf.do({ ~ctrlSet.value( \a, ((~model[ \a ] ? 0) + 0.1.bilinrand).wrap( 0 ,1 )); 0.1.wait })};
    ~rout.stop;

(
    s.waitForBoot({
        ~synth = { var inp, peakPeak, trig;
            inp = AudioIn.ar( 1 );
            trig = Impulse.kr( 20 );
            peakPeak = RunningMax.ar( inp, trig ) - RunningMin.ar( inp, trig );
            SendTrig.kr( trig, 0, peakPeak );
        }.play;
        ~resp = OSCpathResponder( s.addr, [ '/tr', ~synth.nodeID ], { arg time, resp, msg;
            ~ctrlSet.value( \b, msg[ 3 ].clip( 0, 1 ));
        }).add;
    });
)

    // the model continues to work without the GUI:
    w.close;
    ~bang = Updater( ~model, { arg obj, key, val; if( key === \b and: { val > 0.5 }, { "Bang!".postln })});

    ~resp.remove;
    ~bang.remove;

 

VIII. Integrating custom Java GUI Components

If you wish to integrate other java gadgets for which no implementations exists in SuperCollider, there is two approaches: The first one is fast and well suited for presentation-gadgets. Using the JSCPlugView class, you can easily add new components to a window. The limitation here is the missing automatic invocation of action functions. The second approach is to write a proper subclass of JSCView. Often you can use the first approach to prototype that view.

For example, we might want to have a JSpinner component. The functionality of JSpinner is similar to JComboBox (aka JSCPopUpMenu), but it doesn't show a popup menu, instead an up and down arrow allow the user to step through the possible items. A spinner looks like this:

    JSpinnerDenHaag Screenshot

SwingOSC can be used to rather easily script the java language. That is, we can create an manipulate java objects through a proxy on the SuperCollider client side, using the JavaObject class:

    // create an instance of java.awt.Frame
    ~jframe = JavaObject( "java.awt.Frame" );
    // all method calls to the object get forwarded to the
    // server who tries to find the appropriate java method to call...
    ~jframe.setSize( 200, 300 );
    ~jframe.setTitle( "Schnuck" );
    ~jframe.setVisible( true );
    // when we are done, we should destroy the object reference
    // on the server to allow garbage collection
    ~jframe.dispose;  // this is a method in java.awt.Frame! the object still exists!
    ~jframe.destroy;  // this deletes the object reference

    // to return primitive values to SC, append an
    // underscore to the method name. Warning: since the
    // communication with OSC cannot be performed inplace,
    // the method call must be wrapped into a Routine (that's what 'fork' does)!
    //
    // Example: create an instance of java.util.Random
    ~jrand = JavaObject( "java.util.Random" );
    // query a new random value
    fork { ~jrand.nextFloat_.postln }

JSCPlugView simply takes an existing JavaObject and wraps it in a handler that is compatible with JSCView, so you can use it in the regular GUIs:

    ~spinListModel = JavaObject( "javax.swing.SpinnerListModel" );
    ~spinListModel.setList( List[ "Apple", "Pear", "Banana", "Mango" ]);
    ~spin = JavaObject( "javax.swing.JSpinner", nil, ~spinListModel );
    w = JSCWindow.new.front;
    JSCPlugView( w, Rect( 4, 4, 200, 30 ), ~spin );
    ~spin.setValue( "Mango" );

Using the underscore style, you can query the currently selected value:

    fork { ~spin.getValue_.postln };

... but we would rather want to be automatically informed about user actions. We have solved this problem by writing a JSCView subclass that creates an instance of de.sciss.swingosc.ChangeResponder, a helper class that attaches itself to the view and when the user modfies the value, the change is forwarded to SuperCollider via OSC. The ChangeResponder ist created like this:

    JavaObject( "de.sciss.swingosc.ChangeResponder", this.server, this.id, \value )
    

 

Here is the full class:

    // SIMPLE TEST CLASS FOR DEN HAAG SYMPOSIUM !
    JSCSpinnerDenHaag : JSCView {
        var <items, <value = 0;
        
        var acResp;     // OSCpathResponder for change listening
        var model;      // JavaObject of javax.swing.SpinnerListModel
        var changeResp; // JavaObject of de.sciss.swingosc.ChangeResponder
        var spin;       // JavaObject of javax.swing.JSpinner
    
        value_ { arg val;
            value = this.prFixValue( val );
            if( items.size > 0, {
                spin.setValue( items[ value ]);
            });
        }
        
        prFixValue { arg val;
            ^val.clip( 0, items.size - 1 );
        }
    
        items_ { arg array;
            items = array;
            model.setList( items.asList );
        }
    
        prClose {
            model.destroy;
            changeResp.remove;
            changeResp.destroy;
            acResp.remove;
            ^super.prClose;
        }
    
        prInitView {
            var result;
            acResp = OSCpathResponder( server.addr, [ '/change', this.id ], { arg time, resp, msg;
                var newVal = items.indexOfEqual( msg[ 4 ].asString );
                if( newVal.notNil and: { newVal != this.value }, {
                    value = newVal;
                    { this.doAction }.defer;
                });
            }).add;
            spin = JavaObject.basicNew( this.id, this.server );
            model = JavaObject( "javax.swing.SpinnerListModel", this.server );
            result = this.prSCViewNew([
                [ '/local', this.id, '[', '/new', "javax.swing.JSpinner" ] ++ model.asSwingArg ++ [ ']' ]
            ]);
            changeResp = JavaObject( "de.sciss.swingosc.ChangeResponder", this.server, this.id, \value );
            ^result;
        }
    }
    

... and here some test code:

(
    w = JSCWindow.new;
    x = JSCSpinnerDenHaag( w, Rect( 4, 4, 200, 30 ))
        .action_({ arg view; ("Selected index is " ++ view.value ++ "; item is " ++
            view.items[ view.value ]).postln });
    w.front;
)

    x.items = [ "Apple", "Pear", "Banana", "Mango" ];

 

IX. Integrating external Java Classes

To access classes that are not part of the Java SE and which are not in the system class path, you will need to add them to the dynamic class loader, using the addClasses method in SwingOSC. Here is an example for JFreeChart (download from sourceforge.net/projects/jfreechart):

(
    // assuming you have downloaded jfreechart-1.0.6, add these two
    // jars to the class path (replace the dictory with your JFreeChart
    // install dir!)
    x = "file:///Users/rutz/Desktop/jfreechart-1.0.6/lib/";
    g.addClasses( x ++ "jfreechart-1.0.6.jar", x ++ "jcommon-1.0.10.jar" );
)

Now all classes in those two jars should be accessible via SwingOSC. We create a simple pie-chart:

(
    var data, plot, gen;
    data = JavaObject( "org.jfree.data.general.DefaultPieDataset" );
    Dictionary[
        ("Burundi" -> 90),
        ("Ethiopia" -> 110),
        ("Democratic Republic of Congo" -> 110),
        ("Liberia" -> 110),
        ("Malawi" -> 160),
        ("Guinea-Bissau" -> 160),
        ("Eritrea" -> 190),
        ("Niger" -> 210),
        ("Sierra Leone" -> 210),
        ("Rwanda" -> 210.0)]
    .keysValuesDo({ arg key, value; data.setValue( key,value )});
    plot = JavaObject( "org.jfree.chart.plot.PiePlot", nil, data ); data.destroy;
    gen = JavaObject( "org.jfree.chart.labels.StandardPieSectionLabelGenerator", nil, "{0} ({1})" );
    plot.setLabelGenerator( gen ); gen.destroy;
    ~chart = JavaObject( "org.jfree.chart.JFreeChart", nil, "Ten Poorest Countries", JFont( "Helvetica", 24 ), plot, true ); plot.destroy;
)

Now display it, using org.jfree.chart.ChartPanel wrapped into a JSCPlugView:

(
    w = JSCWindow( "JFreeChart", Rect( 200, 200, 560, 440 ));
    JSCPlugView( w, Rect( 2, 2, 556, 396 ),
        JavaObject( "org.jfree.chart.ChartPanel", nil, ~chart ))
        .onClose_({ ~chart.destroy })
        .resize_( 5 );
    JSCStaticText( w, Rect( 2, 400, 556, 36 ))
        .resize_( 8 )
        .align_( \center )
        .string_( "(based on 2004 GNP per capita in US$)" );
    w.front;
)

The result should look similar to this:

     JFreeChart Screenshot