Saturday 22 June 2013

New birth

Finally back from Cannes Lions festival, was doing visual application in collaboration with Arcade Studio and Microsoft for Soft Array (Structure hold an hefty 39 surface pros and 4 all in one Asus).

Software runs real time distributed 3d audio reactive visuals, all synchronized of course.



That project will deserve it's own blog post, I'll do a write up once I'm done

For this project it was the first time I use my new shiny in house tool, called FlareTic.
It's basically written from scratch (almost).

So most people like to explain why they do something, I'm normally not like that (I just tend to say : because it's cool / because I can ;)

For once I have a decent rationale about it, so I decided I would share it :)

So why starting to write your own tool and not using some pre-made framework (like vvvv/openframeworks/cinder) ?


  • I need an executable : original idea is to use it as demo tool, I need to be able to export, that removes vvvv
  • I need source code : I am a programmer, I can read/debug code, and when you have lot of devices any bug in your framework can turn things into hell (when you on crunch time for final installation day/night). So I need to be able to run a full debug ... period.
  • I want to use DirectX : No offense to OpenGL, but their API is clunky. Of course you always end up having an abstraction layer, but starting with already a clean API makes your life easy. So that means for OpenFrameworks or Cinder I need to write a DX rendering layer. Ofx I way too verbose for my taste (so that discards it out of the box), Cinder I already wrote a Directx11 Renderer for it (I should publish it at some point).
  • I want Metro Export : For my phone mostly, that gives point 1 and 3 ;)
  • I want dynamic compilation for shader/script, and some graph editing : Don't want to recompile/copy 43 times an app each time I tweak something. I could use lua and cinder, but dynamic c# compilation is easy and I already got lot of routines in c#, so I might as well reuse them.
  • Showing off your tool and tell you wrote it yourself is cool ;)
Also since I didn't do any gui programming for a while, that would be a good exercise ))

When you start a new tool you have plenty of ideas, organizing them is not as simple. I normally tend to write fast, refine later. I need fast iterations so I can quickly test a new idea, which can make it into the tool and means I adapt rest of the code because it's so awesome, or can just end up in the recycle bin.

First thing you need is some form of gui (code editor window, graph editor, main form).

So I already have a small graph editor, written using Direct2D, which is pretty nice and has all the features I need for it : good font rendering engine, basic shapes, bezier for links, and hit test functions. Please note that hit testers was a bonus, it's plain easy to do, but saves you time since it's already there. So I added undo/select/smart link/zoom/pan features, which where all pretty easy.

Also I had a minimal inspector with basic controls (sliders, toggles...)

Code editor was some simple window with a TextBox. It's was super minimal, but enough for early dynamic compiler tests. So I moved to ICSharpCode.Editor so at least I could have syntax coloring. Intellisense support will be for later (only for the c# version, I don't like it for hlsl, I ditch the file right away anytime I install a new vvvv version ;)

Now last (but not least). Patches were into a TabPanel and scripts were separate windows. That can turn your desktop into a big mess quite fast. So I moved to the awesome DockPanelSuite so I can arrange everything I want Visual Studio style ;)

Here is early version of the tool:


New version (graph didn't changed that much except few refinements, but more on user side) :



Still needs some polish, but already quite nicely useable.

I'll speak about evaluation model in next post (eg: likely today), and two things to finish (and make things clear):

  • This is an in house tool, and I have NO plans to release it in the near future. So please don't ask for a beta answer will be a polite no ;)
  • I am not leaving VVVV (I could see some users to be a bit worried about that). VVVV is still an integral part of my workflow.
  • A decent part of the codebase is shared (vlc player/kinect/dx11 rendering...) so lot of improvements will be beneficial for both softwares at the same time.
  • Some nice new features come from ideas I had with the tool, like the up and coming JSON scene exporter was done so I could design scenes in VVVV and import them in FlareTic. But importer is also in VVVV.




Monday 3 June 2013

Shader Programming and Authoring

For this post i'll go into something a bit different, and speak about things in a more general way.

So we're in 2013, graphics card power has exploded and the "way to go" for real time graphics is shaders.

We see them everywhere, demos, new webgl fancy showcases, visual programming...

Cool thing with shaders (depending on environment you use), is they can easily be compiled on the fly, and you have reflection data to tweak parameters. That can make it an easy "designer friendly tool".

That of course create some issues. Creating bad shaders is easy. Creating good shader subsystem, or good advanced shaders is (very) hard. It's simple to copy paste and modify a random piece of hlsl/glsl/cg found on the net, tweak it to do (approximately) whatever you (kinda) want to do with it. No need for graphics API understanding (almost)

That brings another problem, most of your code base becomes a bulk of copy/paste bits of GPU code (that you don't really know what it does).

That's fine for basic post processors and fancy visuals like in the excellent shadertoy , and for random prototyping, but when you have advanced systems (fully fledged particles systems/geometry processors/compute shaders doing some heavy simulations, then it's a different kettle of fish).

So shader programming is no different than standard programming (in some aspects), you need to really understand how it works, have some coding guidelines and keep code pretty clean (and also know how parallel processing works, specially when you go into gpgpu eg Cuda/Compute/OpenCL).

People will always argue that strict guidelines makes things less tweakable, like having a proper predefined naming, includes and conditional compilation defines to modify shader, pipeline that does one job but very well instead of having a full shader from which you can change any single part if you want, I don't agree with that.

A clean system is easier to maintain, correct naming means all classes use the same base code, without need to change any variable name or semantic, adding properties to data means you only change relevant parts/add a conditional compile define instead of rewrite half of your code base, much easier to run a test compile (less files) and glue things together. It's not easy to do and takes time.

Let's take an example, procedural force field for particles.

Concept is simple as hell, for each particle, get position, compute force based on that position, accumulate to a force buffer (integrate later), or push to velocity right away (not ideal).

Now you can easily have 20+ functions like that. Copying 20 times all the cooking code is prohibitive (not mentioning that if you improve a little part outside of the function you need to copy paste code all the time. That's how most things are done in vvvv and is sucks (badly).

So includes implies some more strict guidelines (naming), but will save you a lot of time (and that's also what I call tweak-able).

For other type of functions (distance functions), you could use the same include in a Sphere Tracer and a particle collider at the same time.

And best of all, you can still use dynamic compilation (using include handlers feature in DirectX).

More advanced (dx11 only), use class linkage and allow several functions in the same handler that you can dynamically switch....

I'll show an example of this at some point, but that's a full topic.

One other thing about shaders is it's a piece of code, but what if you could design shaders on the fly like in vvvv?

For example you have a graph editor in Visual Studio 2012 (it's called DGSL). You can edit your shader via nodes.

Nice isn't it? Not really...

Visual Shader programming is not an easy thing to achieve (as Josh Petrie and other people pointed out in their blog), fitting a feature set which is not too minimal (10 nodes and clicks instead of 2 lines of code), or some nodes which are so complex that you would not need a graph anyway, it's a very hard balance to get.

You mostly end up with a very minimal feature set (like some pixel based material), or have such a low level implementation that it kinda takes twice the time (compared to an efficient code editor with good reflection).


Example 1:
float4 result = a+b;

Why do you need an editor for that? ))

Example 2:
float3 v = cross(v1,v2);

If you don't know what a cross product does, doing it via code or patch will not make any difference.

Example 3:
float3 color = CookTorrance(materialdata); (Patch Cook Torrance with operators)

Patching that with nodes would be a crappy mess, you'd rather have a prebuilt function for it (but then some people will complain they can't tweak...).

Example 4:
float3 color = CookTorrance(materialdata); (Cook Torrance as built in node)

Well just add an include and you go back to example 1.

So in any case (low/high level), I still find it hard to see any decent benefit (not mentioning generally looking at resulting code means any coder will just rewrite it from scratch instead of tweak it).

Also HLSL is still a "minimal" language (compared to java/c#/c++), functions are not that many, no big framework and namespaces, so it's much easier to go into reference page and learn what they do.

So personally I think proper Shader editors with good reflectors, are a much nicer way for authoring.