New Custom Quantization Matrices - EQM AVC Series. Last update: EQM AVC-HR 2005-08-01

Hello Sharktooth, could you elaborate a little on how you deduce your matrices? ("with my eyes" is ok as an answer;) -Skal

Thx. That give me a useful place to start. - Tom

Is there a way to use custom quant matrices with x264 vfw? Edit: Also, is there one similar to MPEG for Xvid? I like the sharpness that MPEG gives... js

MPEG matrix is different from (and more bitrate hungry than) h263. The values i suggested are intender for a h263-like quant2 compression, also i specified those values depends on custom matrices (in both xvid and x264).

Bitrate Variance Question 

Hi Sharktooth, I understand that your AVC matrices are intended for bitrates of around 1350kbps for 720x430 "average motion" movies. From following some of your XviD matrices development, I believe I recall that the matrices are "tuned" for the target bitrate/resolution ratio and perform sub-optimally on lower bitrate/resolution ratios. Getting (slowly) to my question... will the matrices perform sub-optimally in regions of an encode where due to a default curve compression (qcomp = 0.6 in x264) bitrates are reduced because of slow/no motion detection? For instance, I may have an encode which averages 1350kbps but a particular scene may have only 800 kbps allocated due to it being a static scene. Will these matrices perform poorly in such regions solely becuase of the low bitrate or will they still perform as expected since in fact there is little or no motion? The reason I ask is because I am considering unbridling the qcomp by significantly raising the ratetol from its default of 1.0. Thanks for your time!

ColorMatrix 

Hi Sharktooth, I really like your CQM. When I was trying to determing a good bitrate before I used your CQM, I had trouble with blocky dark backgrounds even at what I thought were too high bitrates. When I finally switched to your CQM, I found the blockyness was dramatically reduced. I'm guessing your CQM gives more bitrate in the dark color range than the standard one does. So, all was well in the world. I have always been frustrated by my encodes being a tad too dark though. I recently stumbled onto about ColorMatrix, which corrects that (or the real problem rather). I tried it and in fact it really improved things. So, I started a reencode. Unfortunately, after my first reencode, I found the blockyness was back. Now, there are other differences as well: x264 cli rev 291B instead of 287A, substituted limitedsharpen instead of xsharpen. I've had a hard time finding an old 287a x264 rrev to test with, so I thought I post here and ask if you think the ColorMatrix change could be responsible. Is there a chance that you tuned your CQM for the incorrect color spectrum that ColorMatrix is meant to correct? Not accusing or anything, just asking so I can narrow down the permutations I need to test. Thanks!

Well, limitedsharpen may be the cause. Sharpen filters usually lower the source compressibility in a significative manner. In this situation a lower bitrate matrix is needed but you can try rising the deblocking filter threshold by 1 or 2 to help compensate the compressibility loss. However i dont use colormatrix coz i never experienced darkened or desaturated encodes so IMHO it's not a "tuning" problem.

This matrix will come in handy on my next encodes. I'm doing AVC backups of some high motion cartoon captures @1100kbps and 640x480 resolution, and with this matrix I might even be able to disable deblocking. Once I solve some issues with my current filterchain (SSIQ and aWarpSharp are causing green splotches on the sides of the picture) I'll try this and report back. Sharktooth, how do you expect customs matrices to affect playback compatibility on future AVC capable standalones?