Catflier: Hap Attack (Part 2)

In previous post I explained a bit how to decode Hap files.

I explained a bit how the QuickTime format works, so let's show a bit of code.

First we need to access a leaf node to extract information, for this let's build a small interface:

Code Snippet

public interface ILeafAtomReader
{
    void Read(FileStream ds);
}

Now let's show an example implementation:

Code Snippet

public class ChunkOffsetReader : ILeafAtomReader
{
    private List<uint> chunkOffsetTable = new List<uint>();
 
    public List<uint> Table
    {
        get { return this.chunkOffsetTable; }
    }
 
    public void Read(FileStream ds)
    {
        //Bypass header
        ds.Seek(4, SeekOrigin.Current);
 
        uint entrycount = ds.ReadSize();
 
        for (uint i = 0; i < entrycount; i++)
        {
            uint size = ds.ReadSize();
            chunkOffsetTable.Add(size);
        }
    }
}

This is reasonably simple, we go parse the data we require.

Now there is a little problem, some parsers will read the whole atom, some parsers might only read the data they want, so our file position pointer might not be at the end of the atom.

To circumvent that, let's add a little adapter:

Code Snippet

public class LeafAtomReaderAdapter : ILeafAtomReader
{
    private readonly ILeafAtomReader reader;
 
    public LeafAtomReaderAdapter(ILeafAtomReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException("reader");
 
        this.reader = reader;
    }
 
    public void Read(FileStream ds)
    {
        var currentpos = ds.Position;
        reader.Read(ds);
        ds.Seek(currentpos, SeekOrigin.Begin);
    }
}

This takes another atom reader, but before to let it read, it stores the file position pointer, and restores it once the other reader is done.

Since Atom order is not guaranteed, we also need to tell which containers we are interested in:

Code Snippet

private string[] containers = new string[]
{
    "moov","trak","mdia","minf","stbl"
};

Then once we find the right media, sample table (the one which contains hap), we need to lookup a bit of extra information, so we need to store moov and trak atom offset (so we can then read tkhd to get video size info, and mvhd to get time units).

Code Snippet

if (containers.Contains(fcc.ToString()))
{
    if (fcc.ToString() == "trak")
    {
        this.currenttrakoffset = ds.Position;
    }
    if (fcc.ToString() == "moov")
    {
        this.currentmoovoffset = ds.Position;
    }
 
    //Keep parent position, since we'll want to get this to read sample table
    Parse(ds, ds.Position);
}

Once we found a track with hap, we can jump back to the file position and go read headers.

So now we can finally play hap files.
Only issue, without ssd, this is drive intensive, and we generally have a lot of memory, so let's allow to load the whole video data in ram.

This is done differently in QT and Avi.

In QT I already built the lookup table, so I can just load a copy of the file in memory, and lookup from there:

Code Snippet

public unsafe static DataStream ReadFile(string path, CancellationToken token, IProgress<double> progress, int chunkSize = 1024)
{
    var fs = File.OpenRead(path);
 
    IntPtr dataPointer = Marshal.AllocHGlobal((int)fs.Length);
    IntPtr pointerOffset = dataPointer;
 
    byte[] chunk = new byte[chunkSize];
    int remaining = Convert.ToInt32(fs.Length - fs.Position);
    int read = 0;
 
    while (remaining > 0)
    {
        int toread = Math.Min(remaining, chunkSize);
 
        fs.Read(chunk, 0, toread);
        Marshal.Copy(chunk, 0, pointerOffset, toread);
 
        pointerOffset += toread;
        read += toread;
 
        double p = (double)read / (double)fs.Length;
        progress.Report(p);
 
        remaining = Convert.ToInt32(fs.Length - fs.Position);
 
        if (token.IsCancellationRequested)
        {
            fs.Close();
            Marshal.FreeHGlobal(dataPointer);
            throw new OperationCanceledException();
        }
    }
 
    var ds = new DataStream(dataPointer, fs.Length, true, false);
    fs.Close();
 
    return ds;
}

This is just a simple file reader, that grabs blocks and report progress, so it can be sent as a background task.

For Avi I got no lookup table, but some api to get frameindex -> data (from disk). So I create a memory block large enough to contain the whole video (file size works perfectly for that purpose ;)

Then In background I go request frames and build a prefix sum:

Code Snippet

public unsafe static AviOffsetTable BuildTable(hapFileVFW fileinfo, CancellationToken token, IProgress<double> progress)
{
    long fileLength = new FileInfo(fileinfo.Path).Length;
    int frameCount = fileinfo.FrameCount;
 
    IntPtr dataPointer = Marshal.AllocHGlobal((int)fileLength);
    IntPtr offsetPointer = dataPointer;
 
    List<OffsetTable> offsetTable = new List<OffsetTable>();
 
    int readBytes = 0;
    int currentOffset = 0;
    for (int i = 0; i < frameCount; i++)
    {
        fileinfo.WriteFrame(i, offsetPointer, out readBytes);
 
        OffsetTable t = new OffsetTable()
        {
            Length = readBytes,
            Offset = currentOffset
        };
 
        offsetTable.Add(t);
 
        offsetPointer += readBytes;
        currentOffset += readBytes;
 
        double prog = (double)i / (double)frameCount;
        progress.Report(prog);
 
        if (token.IsCancellationRequested)
        {
            Marshal.FreeHGlobal(dataPointer);
            throw new OperationCanceledException();
        }
    }
    progress.Report(1.0);
    return new AviOffsetTable(offsetTable, dataPointer);
}

This is simple too, we just ask the avi wrapper to write into our pointer, get number of bytes written and move pointer by that offset for next frame. At the same time we build our offset table.

Once we have our data loaded in memory everything is much simpler :

Code Snippet

public IntPtr ReadFrame(int frameIndex, IntPtr buffer)
{
    if (this.memoryLoader != null && this.memoryLoader.Complete)
    {
        var tbl = this.memoryLoader.DataStream;
        IntPtr dataPointer = tbl.DataPointer;
        var poslength = tbl.Table[frameIndex];
        dataPointer += (int)poslength.Offset;
        return dataPointer;
    }
    else
    {
        int readBytes = 0;
        int readSamples = 0;
        Avi.AVIStreamRead(this.VideoStream, frameIndex, 1, buffer, this.frameSize.Width * this.frameSize.Height*6, ref readBytes, ref readSamples);
        return buffer;
    }
}

in first case we just return a pointer from our lookup table (no memory copy required), in the second case we read from disk.

Preloading content into memory gives a huge performance gain (and memory is rather cheap, easy to have 64 gigs in a single machine, so preload can be a definite good option).

So after that comes all the usual cleanup, manage videos element count and make sure we don't have memory leaks / crashes.

Now I have a really nicely working player, why limit our imagination?

First, I wanted to test some 8k encoding, so I exported a few frames from 4v and tried to use virtualdub for encode in hap. Press Save->Out of memory.

So instead, let's just encode directly from vvvv ;)

Writing encoder was easy, you set avi headers with you video size/framerate/compression, then you only need to get texture from gpu, convert in whichever dxt/bc format you want, compress with snappy if required, write frame.

One thing well done!

Next, since we run on dx11 hardware, we have access to new block compression formats:

BC6: three channels half floating point (hdr playback, mmmmmhhhh)
BC7: 4 channels, better quality than BC3/DXT5, but encoding is really slow

So let's add a few more FourCC, and add option in encoder/decoder:

Now we have new hap Formats:

Code Snippet

public enum hapFormat
{
    RGB_DXT1_None = 0xAB,
    RGB_DXT1_Snappy = 0xBB,
    RGBA_DXT5_None = 0xAE,
    RGBA_DXT5_Snappy = 0xBE,
    YCoCg_DXT5_None = 0xAF,
    YCoCg_DXT5_Snappy = 0xBF,
    RGB_BC6S_None = 0xA3,
    RGB_BC6S_Snappy = 0xB3,
    RGB_BC6U_None = 0xA4,
    RGB_BC6U_Snappy = 0xB4,
    RGBA_BC7_None = 0xA7,
    RGBA_BC7_Snappy = 0xB7,
}

Please note that those formats are also available out of the box in OpenGL (btpc compression).
So any software that use GL3.1 + can take advantage of it (and really softwares should already have moved to a GL4+ core profile, so there are NO excuses ;)

Finally, people always tend to think of videos as just a sequence of images.

Although there are some cases where other formats are more suitable (panoramic/dome projection).

In that case cubemaps are much more suited for this.

Oh and DXT/BC formats support cubemap compression.

So let's just write cubemap data as a frame, which was 0 lines of code in my case, since my writer already supports cubemap export.

Then there's only a little twist, in the avi stream info, don't forget to multiply data required by 6 (yes we now have 6 textures in one frame)

in AVISTREAMINFO :
public Int32 dwSuggestedBufferSize;

Is the field where we initiate buffers.

Then decoding frames works exactly like standard textures (Cubemaps are Texture2D too, so loading is done exactly the same way).

There's of course a little twist, in case of CubeTexture we need to set different parameters on ShaderResourceView creation:

Code Snippet

if (videoTexture.Description.OptionFlags.HasFlag(ResourceOptionFlags.TextureCube))
{
    ShaderResourceView videoView = new ShaderResourceView(device.Device, videoTexture);
}
else
{
    ShaderResourceViewDescription srvd = new ShaderResourceViewDescription()
    {
        ArraySize = 6,
        FirstArraySlice = 0,
        Dimension = ShaderResourceViewDimension.TextureCube,
        Format = videoTexture.Description.Format,
        MipLevels = videoTexture.Description.MipLevels,
        MostDetailedMip = 0,
    };
 
    ShaderResourceView videoView = new ShaderResourceView(device.Device, videoTexture,srvd);
}

That's more or less it, cube texture encoding/playback with Hap:

Some days well spent!!

Catflier

Tuesday 28 October 2014

Hap Attack (Part 2)

No comments:

Post a Comment