I spent today hanging out with a friend of mine in San Francisco who does 3D modeling, rendering, and video editing for a living. While not a developer, he's a computer savvy user who can deal with complex user interfaces. After watching him use Adobe Premiere and After Effects for half an our, I did a short demo of PiTiVi. I wish I could say the demo went well, but, in fact, it crashed several times. Talking with him was useful because he has used three major commercial editors: Premiere, Vegas, and Final Cut. He's also familiar with Maya, and After Effects. The interview was casual, so I didn't record or take notes on the interview. A brief summary of what I learned is available here.

Revised Todolist

1. Frame Snapping
2. Collapse Selection
3. Editing Markers (see below)
4. Source Splices
5. Fix Text item clipping
6. Drag multiple selected objects
7. Select Before Current Source
8. Select After Current Source
9. Undo/Redo
10. Multi-layer editing
11. Still Images

1. Make zooming work again
2. Magnetic Timeline Playback Cursor
3. Frame Snapping
4. Collapse Selection
5. Editing Markers (see below)
6. Source Splices
7. Fix Text item clipping
8. Drag multiple selected objects
9. Select Before Current Source
10. Select After Current Source
11. Undo/Redo
12. Multi-layer editing
13. Still Images

Still images are not completely done in the back end, so they are still at the bottom of the stack. I'll bump them up to top priority once they are done in the back end. I am anticipating minor changes in the UI code to make still images work, but we'll see.

I managed to get the razor tool working today, which is incredibly satisfying. Next on the list is to fix zoom support, which was broken by the layout changes. Once that's done, I think it would be nice if the ruler's playback cursor not only extended all the way down the timeline, but was magnetic as well. This would mean that editing operations could easily be made to "snap" directly to wherever the playback head happens to be, including trimming and razoring sources.

Here's my revised todo list:

1. Split clips in advanced UI (Done)
   
2. Make zooming work again (In Progress)
3. Magnetic Timeline Playback Cursor
   
4. Frame Snapping
5. Collapse Selection
6. Editing Markers (see below)
7. Source Splices
   
8. make Text item something we can shrink so it doesn't protrude past edge of clip at high zoom (in progress)
   
9. Drag multiple selected objects
10. Select Before Current Source
    
11. Select After Current Source
12. Undo/Redo
13. Multi-layer editing
    
14. Still Images

I want to introduce a feature I'm calling editing markers. My understanding is that some commercial products do this already, so perhaps some of you will be familiar with this idea. Basically, you should be able to drop an unlimited number of "markers" which will serve a number of functions:

1. visual "bookmark" of some important point in the timeline
2. magnetic edit point for spliting, trimming, and moving sources.
3. general input to editing commands: For example, if you select n edit points and n sources, and then choose the razor tool, a cut will be made at the edit point for each source.

Initial Design

• Place edit points by clicking the marker button, or pressing m. The edit point will appear as a small triangle beneath the ruler at the current playhead position.
  
• You can fine tune edit points simply by draging them
• You can select edit points as you select any other timeline object
• When you activate the razor tool, drag a source, or trim a source, the effective cursor position will snap to the edit marker if the cursor position is within the edit marker deadband.
• If any edit points are selected when the razor tool is active, the razor tool will treat the edit points as input coordinates. Other commands may operate like this as well.
  

See previous posting for current progress. Just wanted to post a screenshot of the new layout:

The orange bar is visible because the razor tool has been activated. Razor tool doesn't actually split the sources yet, but I'll probably get that working tomorrow morning.

Here's my revised todo list:

1. Fix graphic bug created when advanced timeline loads from file which places clips in the wrong order. (Done)
   
2. Layout changes, implement a toolbar(done)
   
3. Delete clips in advanced UI (done)
4. Make zooming work again (=P)
   
5. Split clips in advanced UI (In Progress)
   
6. make Text item something we can shrink so it doesn't protrude past edge of clip at high zoom (in progress)
   
7. make edge snapping work(done)
   
8. Drag multiple selected objects
   
9. Undo/Redo
10. Multi-layer editing
    
11. Still Images

The Vision

After a vigorous discussion on IRC this morning, my goals for the advanced timeline have crystalized in my mind. I'm shifting from my original focus on the simple timeline, and I am going to concentrate on basic editing in the advanced timeline. I thought I'd share my plans, since currently there are no mock-ups for the Advanced Timeline in any form. I wanted to take the time to include some sketches, but I'm getting tired and I want to get my thoughts out.

Design Principles

1. Use direct manipulation for most common operations.
2. Use noun-verb pattern for most other operations: select first, then click a button in the toolbar to perform a command.
   
3. Minimize the number of modes. Modes are evil. Use context to decide which action is appropriate.
   
4. Constrain actions to sensible values by default. Provide modifiers to override these defaults easily.
5. Provide feedback. Change cursor styles, provide real time updates.
   

Most common Tasks

• navigating the timeline
  
• moving sources
• trimming sources
• adjusting edit points

Other Tasks

• deleting sources
• moving large numbers of sources
• grouping multiple sources into single source
• decoupling audio from it's associated video

Layout

• Ruler at the top
• Sources in the middle
• Tools at the bottom
• Tools include: split, select-right, select-left, delete, unlink, group, ungroup,collapse selection,zoom adjustments,
  

Keyboard

• shift, ctrl, alt, and ctrl+alt act as quasi-mode modifiers
  

Design Features

• Only conventional "tool" is the split tool. By default, you activate it first, then click where you want to cut. Later, I want to implement an alternate way of making cuts with markers, but people are already familiar with the current method, and in this case it makes sense to have a mode because it mimics the act of physically cutting a piece of film. After one cut, the tool resets.
  
• Delete, Unlink, Group, Ungroup, Collapse, all operate on current selection. These commands are only sensitive when valid on the current selection.

Selections

• By default, click or click-drag to select.
  
• Use shift to take the intersection of the current selection and the new selection.
  
• Click-and-drag on the current selection will drag every object in the selection.
• Two tool bar buttons will allow the user to extend the selection down the timeline to the right, or to the left, of the currently selected source.

Operations

• Group/Ungroup combine multiple sources into a single source. If the selection includes audio and video, two groups are made (one for audio/video).
• Unlink allows for dissociating linked audio and video tracks.
• Collapse arranges all sources in the current selection so that there are no gaps between them. (perhaps a leftward /rightward variant should be provided)
  

Edge / Frame Snapping

• When moving sources, sources snap to the nearest frame at the clip's native frame rate (depending on zoom), or the project's current framerate if the clip does not have frames. This can be overridden by holding a modifier key. Ctrl will disable frame snapping entirely, while alt will switch to the project's native frame rate.
  
• User can trim a source at any time by clicking-and-draging the trimming handles at the ends of the source widget. A source cannot be stretched beyond its actual duration (time stretching will be implemented later, as one of those evil modes). By default, trimming snaps to the clip's native frame boundaries, but this can be be over-ridden with shift/alt/control as described above with moving sources.
  
• The edit points of sources are "magnetic" within a certain distance. This means it should be really easy to make one source cut directly to another, despite the default frame-snapping. Care must be taken that the snap-to-frame feature doesn't interfere with this feature: it should take priority.

Fine-Tuning of Edit Points

• When the edit point of one source coincides with another, the clips are automatically "spliced", meaning their in/out points become linked together.
  
• When two clips are spliced, the area between the two sources works as an edit-point adjuster. Essentially it adjusts the out point of the left source and the in point of the right source at the same time. A bi-directional arrow cursor <-|-> should be used to indicate the presence of the splice.
  
• By default, edits can snap to the frame boundaries of either clip. The user can override this with alt/control as described above.
  
• The splice can be broken by moving the clips or trimming them as usual: only the area in the center provides this feature.
  
• The UI should constrain the edit point in such a way that neither source can be stretched beyond it's native duration.
• Note this does not address the (possibly more common) scenario in which the user wants to adjust the in / out point of only one of the sources, but still keep the hard cut between them. In this case, the user first performs the trimming operation, then uses the collapse command to eliminate the bank space created.
  

Multi-Layer Editing

• The user can add additional layers to the timeline. For the moment, these will merely be an organizational convenience, but later they will play a role in inputing effect operations.
  
• By default, audio will be mixed, and the topmost video source will play.
  Future releases will introduce compositing support.
  

Seeking

• Scrubbing directly on the timeline ruler will seek to that place in the timeline at the nearest frame in the current project settings.
  
• The user can use forward and back arrows to seek forward/back single frames at the current project framerate.
  
• Holding down keys causes repeated seeking. Holding shift increases the seek interval to 5 frames. Holding down ctrl increases the interval to 1/2 second. holding down alt increases the interval to 5 seconds. Holding down ctrl+alt moves the play head to the next edit point.
  
• If the playhead moves beyond the current scroll position, the timeline scrolls to the keep the playhead position in the center when the user releases a key.

Zooming

• Zooms should always center on the current play head position, not the scroll position. the zoom control should provide meaningful zoom levels: 1, 5, 10 frames, 1, 5, 10, seconds, 1, 5, 10 minutes.
  

My main concern is that I'm relying on modifier keys to control certain parameters: but I think the way I've used them is justifiable. I use them more-or-less consistently, and their function can be explained in a couple of sentences. If the defaults don't turn out to be sane, we can change them, and still use the modifiers to access the other modes. I think only user testing and feedback will really tell us what the defaults should be.

Post-Guadec Update

Ugh. I hate airlines, Delta in particular. Coming home was a real nightmare. I came back to find that my homebrew exploded all over my room, so tomorrow I get to shampoo the carpets. I was only able to salvage about half a pitcher of it, as it had a tendency to leap out of the bottles once the cap was removed. Fortunately, it turned out to be very strong.



I committed changes which implement resizing of clips in the advanced timeline this morning. So there's a pitivi first. I'm tinkering with support for "edge" snapping, which is crucial for usability. Edward is still working on refactoring pitivi core so that we can handle still images. I also added some cosmetic changes, because I was tired of the way things were looking. Unfortunately, the only thing giving you any clue you're resizing something, is the cursor, and that didn't come through in the screenshot.

Here's my revised todo list:

1. Fix graphic bug created when advanced timeline loads from file which places clips in the wrong order.
   
2. Layout changes, implement a toolbar
   
3. Delete clips in advanced UI
4. Split clips in advanced UI
5. make Text item something we can shrink so it doesn't protrude past edge of clip at high zoom
   
6. make edge snapping work
   
7. Drag multiple selected objects
   
8. Undo/Redo
9. Multi-layer editing
   
10. Still Images

I've added Undo/Redo support: now that pitivi can do some dangerous things, undo/redo are crucial. I doubt i'll get to that by the end of the summer, though. Still images are at the bottom of the stack, because at the moment they are impossible to implement. When Edward fixes PiTiVi core, I'll move them up.

In the mean time, i'm thinking about how undo/redo support should work. It's kindof a tricky problem. One hackish way to implement it would be to serialize the timeline to the intermediate format after every major undoable event (via a signal emitted from PiTiVi core or in the UI). If the user wants to undo, then we can simply reload the undo tree, as if we were loading a file. That might be a way to implement single-action undo/redo for the next release. Multiple undo/redo will have to be a little smarter: a stack of operations needs to be mantained somehow, and each operation needs to have an inverse. Undoing in this case means poping the top of the stack, and applying the inverse operation to the timeline. The trick of course, is providing the infrastructure for both maintaining this history, and reversing the changes. Part of the problem is deciding which events are actually undoable. For example, during the course of a drag in the PiTiVi timeline, a source's position will be updated potentially thousands of times. We only care about its position before and after the drag operation, and only the UI can provide this perspective.

I wish I had some kind of handbook on how to implement these big, complicated, yet crucial features.

Guadec Update

Edward and I worked out the following todolist for me:

1. (done) Fix file load/save support
   
2. (done) Multiselection in advanced UI
3. Layout changes
4. Resize clips in advanced UI
   
5. Deleting in advanced UI
6. Split clips in advanced UI
   
7. Multi-selection Dragging
8. Time-stretching of clips

As you can see, the first two items are done. The next step is to make some layout changes to the advanced UI to allow for tools, such as a cutting razor, shifting of sources, what have you. The first tool will be a delete button, which will delete everything in the current selection. Then, I'd like to have a trimming razor which will split the cursor at the mouse position (some kind of floating bar will show where the source will actually be trimmed, an alternative would be to trim at the current playback head position).

Meanwhile, Edward is working on adding still picture support in PiTiVi core.

1. The first step is a bit of refactoring to move useful code into more abstract classes.
   
2. A custom freeze-frame element will be created for actually handling still pictures within a timeline.
   
3. This will be used to implement a TimelineImageSource object in PiTiVi core.
   
4. Finally, changes to the discoverer will be necessary to avoid long delays when importing large images.

So, by the time the PiTiVi backend can handle images, the ComplexTimeline should be a lot more functional. And, finally, you can actually save your work.

Even at GUADEC, Slow Going

GUADEC is a lot of fun, but the network at the venue is a bit dodgy. Worse still, a number of useful ports are blocked. This has complicated just about everything. Also, I tried to update my gstreamer install and the cvs-update.sh script barfed all over everything, borking my cvs install of gstreamer. I might have it fixed by tomorrow morning =(

The good news is that Edward and I fixed a couple of rendering bugs in the advanced timeline. The key to the problem was the GTK+ debugging flag --gtk-debug=updates, which shows you which areas are being redrawn (though I can't yet use this on my machine, still need to recompile the gtk+ library to enable debugging).

Also, I finally got to look at Edward's design notes for the Complex UI. I finally have an idea of what I'm building. There's still a lot to discuss, but Edward and I haven't quite found time for a hacking session.

That's about it for now.

Bugs

Apparently there are some bugs in the latest goocanvas code which cause the timeline to update portions of the display which are much too large. I'll be looking into this at some point this week. Also, a fellow contributor has submitted a patch to add a vertical playhead bar to the timeline.

Update

I lost a couple days due to a stomach bug of some kind (too much street food in Istanbul?) I've been using some of this down time to do research. GUADEC is next week, and I expect that Edward and I will get some serious hacking done during that time.

I've been meaning to interview some of my video professional friends to see how they use existing video editors, and find out what they like and don't like about them. In preparation for that, I have been reading on-line tutorials for existing video editors to get an idea of what expert usage looks like. I like the user submitted tutorials better than official documentation, because it helps me get into the frame of mind of a user, rather than a designer. Tutorial explanations hilight the how users conceive of the problems they are trying to solve much differently than designers. Often they use tools originally intended for a different purpose to achieve their ends. The downside, of course, is that tutorials are mainly written by expert, tech-saavy users. Right now, the main thrust of PiTiVi's development has been targeted at novice users, or at least users unfamiliar with the domain of video editing.

Part of me wonders whether or not this is a mistake: since PiTiVi is a linux video editor, relatively few novice users have access to the application. And we already have Kino, which is more mature than PiTiVi. If I were a novice user trying to get video editing done with Linux, Kino would be my only real choice. On the other end, we have Cinelerra, which seems to offer more high end features (though I've never gotten it to work). Maybe PiTiVi should aim for the middle-of-the road, intermediate crowd. Instead of dumbing down for 5 year olds, or making some ridiculous attempt to out-maneuver FCP (not going to happen), we shoot for an even balance of features: more precision and control than Kino, but don't plaster the screen with controls and widgets. At the same time, we should provide an environment that will be familiar to anyone who has used Premiere, or FCP, or even iMovie, so that their skills transfer. This will fit well with features that gstreamer already provides: an open format timeline, and real time processing of transitions and filters.

PiTiVi could fill a real niche in the amature filmmaking crowd: we can support the odd-ball movie formats that modern digital "still" cameras, like Canon PowerShots, spit out. If PiTiVi makes using these cameras as simple to use as other editors make DV, it will be an attractive tool for people making movies on a shoestring budget.

The existing simple timeline code might be re-imagined as something similar to FCP's media browser: useful for setting rough edit points on clips not yet imported to the timeline, and we needn't worry about providing support for effects or transitions.

Relevant Tasks:

1. clip logging and renaming in the library
2. import movies from "still" cameras
   
3. "off-line" (low-res) editing and recapturing (high-res)
   
4. synchronizing audio and video from different sources
5. animation and timelapse from sequences of stills

For next week, I want to take advantage of the time I have with Edward. He has much more in-depth knowledge of gstreamer and pitivi core, so I'll be taking a break from the UI work to solve some of the back-end problems preventing things like still images from working.

ToDo List:

1. fix the freeze plugin, or write a replacement, so it works with gnonlin
   
2. implement transition and effect objects in PiTiVi core
3. implement still image sources in PiTiVi core
4. discuss the complex UI design (please bring your notes, Edward)
5. (optional) make other mixing modes in videomixer available through property
6. (optional) tinker with gst-editor a bit, see if it can be made more stable