Wednesday, October 29, 2008

Icons Needed

My refactoring of the UI is nearly complete, so I've started adding new features to PiTiVi. Today I got the Unlink/Relink features working. Normally audio and video clips are linked together. Now you can unlink them and manipulate them separately. Relink allows you to reconnect a source to it's mate if you Unlinked them by accident. I have a couple more features planned for the near future, including: "collapse", which eliminates gaps in the selection; and select-before and -after, which select all sources right or left of the currently selected source(s).

This means we'll need some artwork to help represent these features. I'm using stock icons for now, and as you can see, it's not the best fit. The razor tool is easily mistaken for the "cut to clipboard" command, and the it's not at all clear what on earth "connect" and "disconnect" do. Anyone interested?

Master List of Icons

Other things on the horizon include thumbnail previews for sources, and audio waveforms for audio files. After I get through that, I'll start working on a keyframe-curve for audio-volume.

Friday, October 24, 2008

Sorry, it's been a while

Good news! I'm now a Collabora employee, hired part-time to hack on PiTiVi. We have set a deadline for this coming April to finally have a usable release. It's rather ambitious, considering how long this project has been languishing, but I think we're finally in a position to really make some progress. Now that we have basic cutting and trimming, we're moving on to the more challenging features. The following is a list of the user-visible features we want to have working by this April.
  1. moving/trimming multiple items
  2. support for still images (along with zooming, croping, rotating and scaling)
  3. linking and unlinking clips
  4. multi-layer editing...
  5. ..which goes hand-in hand with audio mixing and video compositing...
  6. ...as well as basic video effects.
  7. undo/redo support
This is the big push, and unfortunately, it requires more back-end work. At least this time, I'll be able to help out with a good portion of it, since a few of the changes are based on my own ideas. The refactoring will enable the features listed above, as well as support some long-term goals:
  1. Multi-track editing (i.e. multiple named output channels)
  2. Subsequences and subcompositions
  3. Combining media media from multiple sources into a unified objects (syncing separate audio and video, for example, or combining video from multiple cameras into a single object)
  4. Converting between media types, like converting an audio stream to a video stream with a visualizer plugin.
  5. Pre-rendering and caching of data.
  6. Re-conforming a project between different resolutions, codecs, and container formats.
I have already started work on an improved UI framework, which is almost complete. When the existing interface is ported over to it, we'll be able to add new features more easily than at present.

We're also adding a new UI component: the property editor, which will allow the user to tweak per-object properties. This interface will also be extensible: modules and plugins can provide appropriate controls for a wide range of editing tasks, from chroma key to animation.

We're getting there, slowly but surely.

Saturday, September 13, 2008

Thursday, September 11, 2008

An argument for removing the simple timeline

The problem with the simple and advanced time-line is that the two UIs present different and incompatible abstractions to the user. I'll give you an example. The simple timeline allows for setting clip volume through a slider. Presumably, we want to set a constant volume. The advanced timeline will ultimately allow you to set the clip volume to an arbitrary interpolation curve. These are two different, incompatible representations for volume.

Imagine that a user has spent countless hours tweaking the volume key-frames for a clip. What, then, happens if they switch to the simple time-line. What value does the volume slider display? What happens when the user moves the clip's volume slider? Does it flatten the curve? In that case, should we bring up an obtrusive dialog to warn them against what they are doing?

The easiest solution I can think of is to drop the volume slider from the simple time-line (making the simple time-line much less useful). A better, but harder solution is to keep track of whether key-framing has been enabled on a clip, and if so, disable the volume slider in the simple time-line. This creates extra work. A third solution is to prevent the user from returning to the simple time-line once they've activated certain features, like key-framing. This could be frustrating, though, if the user was exploring and the changes were accidental.

An altogether bizarre solution would be to adjust all the slider values based on the current playback position (through property notifications). The sliders would move up and down during seeking and playback as if moved by ghosts (following the interpolation curve). Moving a slider would essentially create and set a key-frame at the current playback position. This would seem to be an ideal solution at first, because it makes the simple timeline equally as powerful as the advanced time-line. Imagine, though, how frustrating this would be for the user: they set the volume in one part of the clip, seek to another, and the volume has changed. It's not obvious why. So, they move the slider back. The user thinks their changes just didn't take, when actually they're editing a completely hidden data-structure. When they go to preview the video, the audio volume goes up and down erratically.

This is exactly the kind of unnecessary problem caused by splitting the UI. You have one application model and two views. Basically it's pretty much impossible to decouple the view from the model entirely -- the two are going to remain highly coupled because the view is a representation of the model. If you try to support two incompatible views with the same model, your model becomes schizophrenic.

According to the late Jeff Raskin (one of the original designers of the Macintosh), the concept of having a basic and an advanced mode is fundamentally flawed:
Psychologist Clifford Nass....reflects on the common assumption that users can be grouped into two classes: beginners and experts, perhaps with a few temporarily in transition. This dichotomy is invalid. As a user of a complex system, you are neither a beginner nor an expert, and you cannot be placed on a single continuum between these two poles. You independently know or do not know each feature or each related set of features that work similarly to one another. You may know how to use many commands and features of a software package; you may even work with the package professionally, and people may seek your advice on using it. Yet you may not know how to use or even know about the existence of certain other commands or even whole categories of commands in that same package. For example, a user of a photo-processing application who produces only online images may never need, or even learn of, that same application's facility for doing color separations, a feature needed primarily by commercial printers.

Interface designers have tried various approaches to accomodate the premise that usres can be separated into beginners and experts. Because this premise is false, the approaches have failed. Adaptive systems that shift automatically from beginner mode to expert mode when they judge your compentance has reached a certain level is are a good example....On the other hand, there is no theory that tells us that the same fixed interface cannot work well over the full span of a person's experience with it, from novice to old timer. It seems best not to have to shift paradgims during your use of the product, and no elaborate analysis is needed to reveal the advantage of having to learn only one interface to a task.

It is easy to fall into the trap of designing different interfaces for different classes of users because by doing so you can make sweeping assumptions that simplify the design process. Few such assumptions are likely to be true of every user in any reasonably large class of users that you specify. The antidote is to view an interface not from the perspective of a class of users but rather through the eyes of an individual. Every person who uses software over a long period goes through a relatively brief period of learning for each feature or command and a far longer period of routine (and, we hope, automatic) use. We do need to design systems that are easy to learn and understand, but it is more imporant that we make sure that these systems can be efficiently used in the long run.

The learning phase of working with a feature involves your conscious attention. Therefore, simplicity, clarity of function, and visibility are of great importance. The expert phase is predominantly characterized by unconscious use of the feature; such use is enhanced by such qualities as aptness to the task, modelessness, and monotony. These requirements are not in conflict; therefore, a well-designed and humane interface does not have to be split into beginner and expert subsystems. (The Humane Interface, p 68-70)
In other words, if we've done our job right, new users should have no trouble learning the interface (provided they have sufficeint knowledge of video editing). We need to strive to make things clear, simple, and visible. I don't know how much clearer you can get than a proportional timeline in which the length of a clip is proportional to its duration. I don't know how much simpler you can get than click-and-drag to move or resize the clip. We make things more visible by putting all the features of the program out in the open, particularly key-frames, where users can see them. And this system works for advanced users too: by focusing on noun-verb interaction and drag-and-drop, we eliminate modes, and provide monotony, allowing users to get comfortable with the command set.

Wednesday, September 10, 2008

Future Plans

This is my personal wish-list of PiTiVi development. This represents my goals for the next 4 months or so of development, assuming a lazy pace:
  1. Deprecate the Simple Timeline, or at least scale back it's scope so that changes to the simple timeline cannot do much damage in the advanced timeline.
  2. Add seeking previews during drag operations
  3. Rudimentary undo support
  4. Editing markers
  5. Still Images
  6. Still Image Sequences
  7. Design Undo/Redo framework
I'd really like to bring some new developers on board. If anyone wants to work on this stuff with me (particularly undo), I'll help you get up to speed on the code base.

The problem with the simple and advanced time-line is that the two UIs present different and incompatible abstractions to the user. For example, imagine that a user has spent countless hours tweaking the volume key-frames for a clip. What, then, happens if they switch to the simple time-line and move the clip's volume slider? Does it flatten the curve? What value does the volume slider display? The easiest solution to this problem is to drop the volume slider from the simple time-line (or at least disable it if key framing is enabled.

Monday, September 8, 2008

Guadec Wrap-Up

Guadec is officially over now. I have taken the final step of uploading a tar ball to Google's website (this is a formality). The work from SoC has been merged into trunk, but unfortunately problems have materialized in testing. School has started for me which means I am scaling back my commitment of time. I'm hoping to hear from new developers who want to pick up where I left off. A couple of things need to be more fully specified:
  1. Undo/Redo support (which operations will support it), and how to implement it
  2. Back end support for effects and transition
In some respects I didn't get all that far this summer. I started out by trying to simply port both timelines over to goocanvas, and this job consumed the majority of my time. Image support is still lagging, though we are a bit closer to implementing this.

I need to update my design document a bit: I actually want to make the UI a little more general. Instead of a "content-image" portion of each timeline object, I just want to have a "UI" portion. I want to be able to define custom UI for different kinds of media and effects that will appear within each box. The default "UI" will be a keyframe editor, but other UI modules could be constructed. The trick will be in choosing which UI will appear when. Perhaps a factory-based system could be used, mapping factory types to their appropriate UI. This would be good for plugins that want to define new source/effect types and then have UI for them.

The simple timeline broke somewhere between the time I stopped working on it and the end of the summer. I have no idea what's going on, but it's damned annoying. I'm kindof not that interested in maintaining the simple timeline anymore. I'd much rather focus on the advanced timeline, because really, advanced isn't that hard to deal with. The goal of the simple timeline was to be easy to implement, IMHO, and it's actually a little harder to work on than the advanced timeline at this point. But in any case, being able to switch back and forth is going to create complications that I don't really want to have to worry about.

Edward is working on switching PiTiVi and gstreamer over to git (from svn). When that happens I'll start maintaining my own branch where I can hack on some things a little more freely. In the mean time, my future commits will probably be related to bug fixes so that people can actually use the work that I did this summer.

Finally, There's one new feature I wanted to add to make the minimal feature set that the advanced timeline now support a little bit nicer: When you trim a source or drag a source in the timeline, I want the viewer to seek to the relevant edit point. I.e. when you move a source, you should see the point in the timeline just before where that source comes in in the viewer. When you trim a source, you should see the exact in-point that you have set in the viewer. All of this should happen while you are dragging the mouse so that you get instant feedback on your editing decisions.

Wednesday, August 13, 2008

Choices

I've been working on the design a bit more. The top part of the diagram is an anatomical description of what objects in the advanced timeline should look like. Ideally I'd like all objects to support keyframeable properties, not just effects. This will require some changes to PiTiVi core -- in particular, all timeline objects will have to support getting/setting keyframes, which means internally they'll all have gst-controller objects. Personally, I think that's fine: one thing people overwhelmingly seem to want is to be able to directly control image opacity and audio volume.

One thing I'm not quite sure how to handle, though, are simple effects. You see, Edward's paper notes suggest that simple-effects (effects which only take one source as input) should appear in a small gutter above the other timeline objects within a given layer. This gutter would also be used for transitions. But this design leaves no room for over-laying keyframes, unless we allow simple-effects to expand to full height. But if we do that it will be difficult to distinguish beween simple effects and complex effects. So, perhaps *all* effects should simply be full-size objects , as in the above design. One thing I don't like about this, though, is that it's actually kinda hard to tell what's going on. Will videobox apply to just the image source directly below it, or the entire composition?

So, I came up with a different design. Move the properties down below the timeline, but keep them in sync with the timeline's scroll position. Now all objects can have as many properties as they want, and they can be kept completely separate from each other (or they could show as overlapping curves, I'm not yet sure which is best. The idea is that you select an object (in this image, the lighter colored simple effect is selected), and the property viewer displays its property graph. If you select multiple objects, one of two things could happen: it could show the intersection of the properties of all selected objects and let you set the same keyframe/values for all selected objects; or the property browser could display the properties of all selected objects individually. Ideally you'd want some combination of both, but I can't think of an elegant way to do that just yet.

Details: In either design, whenever you adjust a property, the viewer should show a preview of the keyframe after the effect has been applied.