Sunday, 10 August 2014

Implicit Converters

I already spoke a bit about converting types.

Since in FlareTic all the "dataflow patch types" use some form of generic holder for pins, link is done simply as:

Check if input type is assignable from output type, if yes allow link.
Now this creates some issues as for example float is not assignable to bool.

So I don't want to create AsBool (Float) , AsFloat(Bool) and so on for all types, that makes a patch bloated for nothing, I need another solution.

Enters converters

Code Snippet
  1. public interface ITypeConverterFactory
  2. {
  3.     bool CanConvert(Type from, Type to);
  4.     ITypeConverter CreateConverter();
  5. }
  7. public interface ITypeConverter
  8. {
  9.     object Convert(object from);
  10.     bool Warning { get; }
  11.     string WarnMessage { get; }
  12. }

The first interface allows to check if we can convert types, and when we connect creates the converter that does the job.

Now when we check for connection, we have a list of factories, so if we find a converter, we can link :)

For raw .NET primitive types, you have to build by yourself, so to speed this up, few generic builders:

Code Snippet
  1. public class FuncConverter<TFrom, TTo> : ITypeConverter
  2. {
  3.     private readonly Func<TFrom, TTo> convertFunc;
  4.     private readonly bool warning;
  5.     private readonly string message;
  7.     public FuncConverter(Func<TFrom, TTo> func, bool warning, string message)
  8.     {
  9.         this.convertFunc = func;
  10.         this.warning = warning;
  11.         this.message = message;
  12.     }
  14.     public object Convert(object from)
  15.     {
  16.         return this.convertFunc((TFrom)from);
  17.     }
  19.     public bool Warning
  20.     {
  21.         get { return this.warning; }
  22.     }
  24.     public string WarnMessage
  25.     {
  26.         get { return this.message; }
  27.     }
  28. }

And the factory :

Code Snippet
  1. public abstract class FuncConverterFactory<TFrom, TTo> : ITypeConverterFactory
  2. {
  3.     private readonly Func<TFrom, TTo> convertFunc;
  4.     private readonly bool warn;
  5.     private readonly string msg;
  7.     public FuncConverterFactory(Func<TFrom, TTo> func, bool warning = false, string message = "")
  8.     {
  9.         this.convertFunc = func;
  10.         this.warn = warning;
  11.         this.msg = message;
  12.     }
  14.     public bool CanConvert(Type from, Type to)
  15.     {
  16.         return from == typeof(TFrom) && to == typeof(TTo);
  17.     }
  19.     public ITypeConverter CreateConverter()
  20.     {
  21.         return new FuncConverter<TFrom, TTo>(this.convertFunc, this.warn, this.msg);
  22.     }
  23. }

Next bit of boilerplate, could have done several ways, but you only do it once ;)

Code Snippet
  1. public class IntToFloatConverter : FuncConverterFactory<int, float>
  2. {
  3.     public IntToFloatConverter() : base(i => i) { }
  4. }
  6. public class FloatToIntConverter : FuncConverterFactory<float, int>
  7. {
  8.     public FloatToIntConverter() : base(f => (int)f, true, "Conversion from float to int is explicit, some data can be lost" ) { }
  9. }

You can notice that explicit casting also induce a message, so we can warn user in some cases.

Now for all other types, we go reflect implicit/explicit table for other types:

Code Snippet
  1. public class ImplicitReflectionConverterFactory : ITypeConverterFactory
  2. {
  3.     public bool CanConvert(Type from, Type to)
  4.     {
  5.         var f = from.GetMethods().Where(
  6.             m => m.Name == "op_Implicit"
  7.             && m.ReturnType == to
  8.             && m.GetParameters().Count() == 1
  9.             && m.GetParameters()[0].ParameterType == from).FirstOrDefault() != null;
  11.         if (f) { return true; }
  13.         f = to.GetMethods().Where(
  14.             m => m.Name == "op_Implicit"
  15.             && m.ReturnType == to
  16.             && m.GetParameters().Count() == 1
  17.             && m.GetParameters()[0].ParameterType == from).FirstOrDefault() != null;
  19.         return f;
  20.     }
  22.     public ITypeConverter CreateConverter()
  23.     {
  24.         return new ImplicitReflectionConverter();
  25.     }
  26. }

Now let's go a bit forward, why not also being able to auto convert opencv image to dx11 texture, that's why you use a converter factory, since now converter will also hold a resource, so we need one instance per link.

Converter code is just boilerplate, create texture for the same format as opencv image, and copy content in gpu when possible.

Here we go, of course user interface must reflect that we are using a converter, so blue mean converter, yellow means converter with warning.

Now opencv image can be directly linked to any node that wants a texture, and you can notice sharpdx types can be auto translated (Color4 has a Vector3 explicit operator, so it's automatically scanned by the reflector).

I love clean patches ;)

Now one common issue, we might want some form of control over the conversion, for example, float to string, we'd like to specify a format. So let's push the concept forward and make our link configurable via inspector.

Easy then, let's add a small additional interface:

Code Snippet
  1. public interface IConfigurableTypeConverter : ITypeConverter
  2. {
  3.     IEnumerable<IParameter> Parameters { get; }
  4. }

And here we go:

Code Snippet
  1. public class StringFormatConverter : IConfigurableTypeConverter
  2. {
  3.     private StringParameter format;
  5.     public StringFormatConverter()
  6.     {
  7.         this.format = new StringParameter(new StringParameterConverter());
  8.         this.format.Value = "f";
  9.         this.format.Name = "Format";
  10.     }
  12.     public object Convert(object from)
  13.     {          
  14.        return String.Format("{0:" + format.Value + "}", from);
  15.     }
  17.     public bool Warning
  18.     {
  19.         get { return false; }
  20.     }
  22.     public string WarnMessage
  23.     {
  24.         get { return ""; }
  25.     }
  27.     public IEnumerable<IParameter> Parameters
  28.     {
  29.         get
  30.         {
  31.             yield return this.format;
  32.         }
  33.     }
  34. }

Now when we select a link in the editor, if it implements this interface we can manage properties, as per the screenshot:

So far it's great, using the reflection from c# helped to speed up the process a lot.

On next stage, we have the obvious question/problem, what happens if we have multiple converters that allow similar types?

Here we have several choices:
  • Take the first one
  • Show a selector when we complete link
  • Have the selector in the inspector instead
I haven't made a choice, but take first one + selector is the most sensible option for now.

Sunday, 3 August 2014

Timeliner - Part 2

So after playing with doing dx11 track rendering, I was speaking with some friends, and they shown me some screenshots of after effects timeline (I knew a bit how it was looking like but my memory was rusty).

What I actually like in the is the minimalism, you don't really display curves, which is pretty cool, and then all timelines are consistent, it makes it much easier to use several data types (and in my tool i got a lot).

That would also integrate pretty well with my tool, since I can easily push a parameter to the timeline, track count will grow rather steadily.

So I start to work on the design, since I got a Direct2d renderer I might as well reuse is (it's far from perfect, but it's nice enough and I can add improvements on the way, why rewrite something from scratch again anyway).

I decided to take group into account right away, and just draw circles instead of rotated quads (was lazy to build path, and anyway circle look nice).

Instead of directly integrating I just generate a small dataset so I can have a feel,

First iteration is nice, now you of course need a header and a time ruler, plus play/stop/loop buttons.

Next you need to show information about a keyframe, so I built a small tooltip:

Now I wrote track evaluators, to start to attach tracks to parameters, add a register (so if a parameter datatype is not registered I simply can't create a track from it.

Since speaking about evaluation, I looked at 3 techniques:

  • Check in linear way: You have a list of keyframe sorted by time, and search from the beginning. This is obviously the easiest, but the further you are in your timeline the slower it gets. If you don't have many tracks/keyframes it's ok, but it can become inefficient rather fast. This is how it's done in 4v timeline so far.
  • Linear playback optimization: Since you consider that most time you will just play, you store current key frame, and on each time step check if you passed the next. This is how it's done in Duration. This is totally great for pure linear playback, but you need to fall back to the linear version for seeking. If you start to add timeline automation you can have some issues.
  • Tree based structures : You use whatever form of binary tree optimization, so you get faster look up. It costs a bit more on adding/removing keyframe, but you get a decent gain on playback (which is generally desired in case of real time rendering). In my case I use interval tree, which is a technique I used first on databases 10 years ago, and used it to optimize some skinning animations a while back too. It's easy to implement, and you get a balance between linear playback/seek, which fits pretty well my use case. I could also easily fallback to previous technique if I need automation, or even better do a mix of both (use tree as soon as you need a seek, then move back to linear optimized version once you got your keyframe again).

Next, I'm not interested to build groups myself, so they simply are generated per patch, which makes my life much easier, and it's much clearer to see what parameter belongs where:

Next you do all the usual part, standard editor features, probably forgot same that are done, but well as a user you know what's required ;)
  • Load/Save to file (json)
  • Make collapsing work
  • Move key-frames
  • Move global timer
  • Add a node to retrieve timer from timeline
  • Add more datatypes (for now there's bool/float/int/vector2/vector3/quaternion/string)
To integrate in my tool it's also pretty simple, press shortcut on a parameter to create a track, then another shortcut to push value from it at current time, simplicity at it's best.

After one thing , you also want to edit values, so I found that showing editor windows was cumbersome (but needed for some types, so still in the plan).

So for some simple types I added a small shortcut handler, so if a keyframe is selected you can do quick keyboard control (like arrows for value, space switches a bool on/off).

Still some features missing of course (multi select is next but easy as hell, small editor widgets), but even in that state it's already pretty usable, which I find great considering I did not even work more than 2 days on it, really it's impossible to do a timeliner wrong ;)

I specially love how nicely it integrates, push parameter, add key-frames, workflow feels pretty seamless it's great.

Here is a sample of the whole layout in action (visual suck but it's to show the timeline after all)

New milestone reached ;)