Quantcast
Channel: Intel® C++ Compiler
Viewing all articles
Browse latest Browse all 1616

Optimization bug which can lead to access violation

$
0
0

Hello!

I compiled the x264 project with Intel Compiler 15.0, windows, x86, and found the buggy-generated code at O2/O3 optimization level.

static float predict_row_size( x264_t *h, int y, float qscale )
{
    x264_ratecontrol_t *rc = h->rc;
    float pred_s = predict_size( &rc->row_pred[0], qscale, h->fdec->i_row_satd[y] );
    if( h->sh.i_type == SLICE_TYPE_I || qscale >= h->fref[0][0]->f_row_qscale[y] )
    {
        if( h->sh.i_type == SLICE_TYPE_P&& h->fref[0][0]->i_type == h->fdec->i_type&& h->fref[0][0]->f_row_qscale[y] > 0&& h->fref[0][0]->i_row_satd[y] > 0&& (abs(h->fref[0][0]->i_row_satd[y] - h->fdec->i_row_satd[y]) < h->fdec->i_row_satd[y]/2))
        {
            float pred_t = h->fref[0][0]->i_row_bits[y] * h->fdec->i_row_satd[y] / h->fref[0][0]->i_row_satd[y]
                         * h->fref[0][0]->f_row_qscale[y] / qscale;
            return (pred_s + pred_t) * 0.5f;
        }
        return pred_s;
    }
    else
    {
        float pred_intra = predict_size( &rc->row_pred[1], qscale, h->fdec->i_row_satds[0][0][y] );
        return pred_intra + pred_s;
    }
}

Inspecting the generated asm, you can see that "h->fref[0][0]->i_type" value is read from memory (for the purpose to store it on stack) BEFORE executing "h->sh.i_type == SLICE_TYPE_I" condition. But "h->fref[0][0]" is NULL when "h->sh.i_type == SLICE_TYPE_I", which leads to access violation.

I can publish more details about build parameters and command line for resulting executable run with crash, if need.


Viewing all articles
Browse latest Browse all 1616

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>