Android平台aac谷歌软解框架和流程、解码库学习

前言 ：

      在Android系统4.1上面目前aac谷歌软解已经给了两套方案，一套是沿用以前的解码库，一套是使用Fraunhofer Institute开发出来的aac解码库，目前谷歌已经切换到了后者，Fraunhofer提供了一套标准的调用接口，谷歌实现的aac软解component逻辑类SoftAAC2.cpp只要按照这个标准去调用，就能完成aac的解码。

正文

一：SoftAAC2解码流程

前面已经有博客分析了android中是如何加载解码组件的，这里不在重点看android framework中的omxcodec类是如何控制解码组件完成各种解码中的工作，这里默认已经加载好了omx.google.aac.decode这个组件，看看SoftAAC2.cpp中是怎样完成所有逻辑的，跟着打出来的log进行跟踪

01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 constructor....

首先会进入SoftAAC2的构造函数

01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 initPorts....

为这个解码器初始化两个端口（输入和输出，portIndex分别为0和1），看下port对应的结构体：

struct PortInfo {

OMX_PARAM_PORTDEFINITIONTYPE mDef;

Vector mBuffers;

List mQueue;

enum {

NONE,

DISABLING,

ENABLING,

} mTransition;

};

port对应的结构体中包含了三部分的内容

1：关于port所有信息的结构体OMX_PARAM_PORTDEFINITIONTYPE，定义在omx_core.h标准中，需要好好看下；

2：每个port都会对应分配几块buffer（对应的结构体为bufferInfo，后面详解），用于放置解码前后的数据

3：port也维护了一个状态

初始化过程就是初始化OMX_PARAM_PORTDEFINITIONTYPEdef变量，最后调用addPort加入到一个容器中去管理

void SimpleSoftOMXComponent::addPort(const OMX_PARAM_PORTDEFINITIONTYPE &def) {

CHECK_EQ( def.nPortIndex, mPorts.size());

mPorts.push();

PortInfo *info = &mPorts.editItemAt(mPorts.size() - 1);

info->mDef = def;

info->mTransition = PortInfo::NONE;

}

01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 initDecoder before call aacDecoder_open...

01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 initDecoder before call aacDecoder_GetStreamInfo...

其次是在initDecoder函数初始化解码器，调用aacDecoder_Open接口获得解码库的句柄mAACDecoder，以及调用aacDecoder_GetStreamInfo获取aac的一些信息，保存在结构体CStreamInfo中。

mAACDecoder = aacDecoder_Open(TT_MP4_ADIF, 1);

mStreamInfo = aacDecoder_GetStreamInfo(mAACDecoder);

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocating 4 buffers of size 8192 on input port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x40069f48 on input port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x401fd3b8 on input port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x400a7008 on input port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x400a70c0 on input port

为输入端口分配四个buffer（个数是在初始化port过程中kNumInputBuffers指定的），这四个buffer的作用就是让omx client往里面放原始的aac数据，并且调用emptyBuffer，告诉解码器，将这些buffer拿去解码，这些buffer都是通过共享内存实现的，也就是omx client和omx component共享这段内存，client往里面写，component读取里面的数据，看下buffer对应的数据结构：

struct BufferInfo {

OMX_BUFFERHEADERTYPE *mHeader;

bool mOwnedByUs;

};

包含了两部分内容：

1：buffer中包含的数据的结构信息OMX_BUFFERHEADERTYPE，定义在openmax头文件中；

2：buffer当前被谁占有，client(mOwnedByUs为false)或者component(mOwnedByUs为true)。

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocating 4 buffers of size 16384 on output port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x401fe6a8 on output port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x404d09c8 on output port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x408bedd8 on output port

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x408bef00 on output port

同理，为输出端口分配四块buffer，用于client和component共享，这四个buffer用于存放解码器解码后的pcm数据，解码器往里面写，omx client读取里面的数据，其他和input port对应的buffer一致。

上面八个buffer都是通过android中Shared Memory机制实现，内存是在OMXCodec中分配的

size_t totalSize = def.nBufferCountActual * def.nBufferSize;

mDealer[portIndex] = new MemoryDealer(totalSize, "OMXCodec");

for (OMX_U32 i = 0; i < def.nBufferCountActual; ++i) {

sp mem = mDealer[portIndex]->allocate(def.nBufferSize);

CHECK(mem.get() != NULL);

......

}

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent GetParameterWrapper..

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent getParameter ....

01-07 01:39:14.270 E/SoftAAC2(  151): huangbangbang in SoftAAC2 internalGetParameter index is 33554433

01-07 01:39:14.270 E/SoftAAC2(  151): huangbangbang in SoftAAC2 internalGetParameter in default...

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent internalGetParameter....

01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in internalGetParameter in case OMX_IndexParamPortDefinition....

01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent msgType = 0 (kWhatSendCommand: 0, kWhatEmptyFillThisBuffer: 1, kWhatFillThisBuffer:2)

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent onSendcommand.

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent onSendcommand in onChangeState.

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent onChangeStage state=2, component_cur_state=1

01-07 01:39:14.280 E/SoftOMXComponent(   151): huangbangbang in SoftOMXcomponent notify..

01-07 01:39:14.280 V/OMXCodec(   151): [OMX.google.aac.decoder] onStateChange 2

01-07 01:39:14.280 V/OMXCodec(   151): [OMX.google.aac.decoder] Now Idle.

01-07 01:39:14.280 E/SoftOMXComponent(   151): huangbangbang in SoftOMXcomponent SendCommandWrapper..

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent sendCommand....

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent msgType = 0 (kWhatSendCommand: 0, kWhatEmptyFillThisBuffer : 1, kWhatFillThisBuffer:2)

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent onSendcommand.

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent onSendcommand in onChangeState.

01-07 01:39:14.280 E/SimpleSoftOMXComponent(   151): huangbangbang in SimpleSoftOMXComponent onChangeStage state=3, component_cur_state=2

01-07 01:39:14.280 E/SoftOMXComponent(   151): huangbangbang in SoftOMXcomponent notify..

01-07 01:39:14.280 V/OMXCodec(   151): [OMX.google.aac.decoder] onStateChange 3

01-07 01:39:14.280 V/OMXCodec(   151): [OMX.google.aac.decoder] Now Executing.

上面的log表示，client调用getParameter和setParameter获取或者设置解码器的一些参数，changeState改变解码器当前状态等函数，解码器本身也维护了一个状态机，每个状态下面需要或者不能做哪些事情是有规定的，可以在openMAX文档中查看，如下图，

openMAX中定义的component的状态机

从log中可以看出，我们的aac解码器状态机的变化是：

UNLOADED------->LOADED------->IDLE--------->EXECUTING

到达EXECUTING状态之后，就能够开始读写数据编解码了

谷歌实现的软解码的逻辑类采用了多态机制，因为谷歌继承的不只是aac decoder一个component，还有很多其他类型文件的解码器，所以为了达到代码的最大共用，谷歌实现先了两个父类，对于和具体解码器无关的一些操作都放在父类中完成，继承关系如下：

01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled.....

01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled in portIndex is 0 mInputBufferCount == 0.....

01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled in portIndex is 0 before call aacDecoder_ConfigRaw.....

解码器，输入输出端口，每个端口需要的buffer都分配好了，下面就是真正调用解码库进行数据的解码了，真正音频数据前面都会有一些configure数据，所以来到的第一个input buffer我们没有去调用aacDecoder_DecodeFrame函数直接去解码，而是调用 aacDecoder_ConfigRaw  去重新获取aac的一些信息，如sampleRate和numChannels，保存在CStreamInfo结构体中，如果sampleRate和numChannels参数有变化，可能还需要disable output port，并且重新初始化output port和对应的四个buffer，对于CStreamInfo结构体将在第二部分讲解解码库的时候进行

01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled.....

01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled in while looper.....

01-07 01:39:14.460 E/SoftAAC2(  151):  in onQueueFilled in while before call aacDecoder_Fill and aacDecoder_DecodeFrame bytesvalid is 416

所有准备工作都做好了之后就开始真正aac数据的解码了，解码都是在onQueueFilled函数中进行，剔除一些参数的初始化，主要过程就是下面的while循环：

AAC_DECODER_ERROR decoderErr = AAC_DEC_NOT_ENOUGH_BITS;

while (bytesValid[0] > 0 && decoderErr == AAC_DEC_NOT_ENOUGH_BITS) {

aacDecoder_Fill(mAACDecoder,

inBuffer,

inBufferLength,

bytesValid);

decoderErr = aacDecoder_DecodeFrame(mAACDecoder,

outBuffer,

outHeader->nAllocLen,

0 );

if (decoderErr == AAC_DEC_NOT_ENOUGH_BITS) {

ALOGW("Not enough bits, bytesValid %d", bytesValid[0]);

}

}

要明白while循环里面aacDecoder_Fill和aacDecoder_DecodeFrame的作用，需要明白解码库的buffer机制：

解码过程中除了前面已经分配的input buffer和output buffer，解码库还会分配一个过渡性的decoder-internal input buffer，这个buffer大小又RANSPORTDEC_INBUF_SIZE规定，可以任意设定但必须满足两个条件：

1：each input channel requires 768 bytes

2：the whole buffer must be of size 2^n

So for example a stereo decoder:

TRANSPORTDEC_INBUF_SIZE = 2 768 = 1536 => 2048（选择2048bytes）

aacDecoder_Fill就是从input buffer往ecoder-internal input buffer里面拷贝数据，返回的是input buffer中还剩下多少没有被拷贝的数据，如果input buffer中的数据全部拷贝完毕了，就需要re-fill，下图是input buffer的变化图（from aacDecoder.pdf）

aacDecoder_DecodeFrame用来解码internal buffer中的数据，如果数据不足以解码，则返回AAC_DEC_NOT_ENOUGH_BITS，继续调用aacDecoder_Fill填充数据。

这样解码一直进行下去，知道OMXCodec中读取aac原始数据过程中发现已经到该track的结尾，就会给这个mediabuffer打上一个标签，在onQueueFilled函数中检测的这个flag，解码完这一帧之后，就停止解码，并且做所有的release操作

OMXCodec::drainInputBuffer

if (signalEOS) {

flags |= OMX_BUFFERFLAG_EOS;

} else {

mNoMoreOutputData = false;

}

SoftAAC2::onQueueFilled

if (inHeader->nFlags & OMX_BUFFERFLAG_EOS) {

......

}

最后就是free所有的buffer，disable端口等

二：解码库学习   Advanced Audio Coding DecoderLibrary

1，术语：

MPEG-2 and MPEG-4

AAC Low-Complexity (AAC-LC),

High-Eficiency AAC v2 (HE-AAC v2),

AAC Low-Delay (AAC-LD), andAAC Enhanced Low-Delay (AAC-ELD)  decoder

2，主要头文件：#include

3，常用的数据和数据结构：

struct CStreamInfo

This structure gives information about the currently decoded audio data. All fields are read-only, public attributes are as follows:

· INT sampleRate

· INT frameSize

· INT numChannels

· AUDIO_CHANNEL_TYPE pChannelType

· UCHAR pChannelIndices

· INT aacSampleRate

· INT profile

· AUDIO_OBJECT_TYPE aot

· INT channelConfig

· INT bitRate

· INT aacSamplesPerFrame

· AUDIO_OBJECT_TYPE extAot

· INT extSamplingRate

· UINT flags

· SCHAR epConfig

· INT numLostAccessUnits

· UINT numTotalBytes

· UINT numBadBytes

· UINT numTotalAccessUnits

· UINT numBadAccessUnits

Defines:

· #define IS_INIT_ERROR(err) ( (((err)>=aac_dec_init_error_start) && ((err)<=aac_dec_init_-

error_end)) ? 1 : 0)

· #define IS_DECODE_ERROR(err) ( (((err)>=aac_dec_decode_error_start) && ((err)<=aac_dec_-

decode_error_end)) ? 1 : 0)

· #define IS_OUTPUT_VALID(err) ( ((err) == AAC_DEC_OK) jj IS_DECODE_ERROR(err) )

· #define AACDEC_CONCEAL 1

· #define AACDEC_FLUSH 2

· #define AACDEC_INTR 4

· #define AACDEC_CLRHIST 8

Typedefs:

      typedef struct AAC_DECODER_INSTANCE HANDLE_AACDECODER

Enumerations:

      enum AAC_DECODER_ERROR

      enum AACDEC_PARAM

Functions:

· LINKSPEC_H AAC_DECODER_ERROR  aacDecoder_AncDataInit (HANDLE_AACDECODER  self, UCHAR buffer,                                                                                  int size)

Initialize ancillary data buffer.

· LINKSPEC_H AAC_DECODER_ERROR  aacDecoder_AncDataGet (HANDLE_AACDECODER  self, int index,                                                                               UCHAR ptr, int size)

Get one ancillary data element.

· LINKSPEC_H AAC_DECODER_ERROR  aacDecoder_SetParam (const HANDLE_-AACDECODER self, const                                                       AACDEC_PARAM param, const INT value)

Set one single decoder parameter.

· LINKSPEC_H AAC_DECODER_ERROR  aacDecoder_GetFreeBytes (const HANDLE_-AACDECODER self, UINT                                                                      pFreeBytes)

Get free bytes inside decoder internal buffer.

· LINKSPEC_H HANDLE_AACDECODER  aacDecoder_Open (TRANSPORT_TYPE trans-portFmt, UINT                                                                                   nrOfLayers)

Open an AAC decoder instance.

· LINKSPEC_H AAC_DECODER_ERROR  aacDecoder_ConfigRaw (HANDLE_AACDECODER  self, UCHAR conf[ ],                                                             const UINT length[ ])

Explicitly configure the decoder by passing a raw AudioSpecificConfig (ASC) or a StreamMuxConfig (SMC),  contained in a binary buffer. This is required for MPEG-4 and Raw Packets file format bitstreams as well  as for LATM bitstreams with no in-band SMC. If the transport format is LATM with or without LOAS,

configuration is assumed to be an SMC, for all other file formats an ASC.

· LINKSPEC_H AAC_DECODER_ERROR  aacDecoder_Fill (HANDLE_AACDECODER self,

UCHAR pBuffer[ ], const UINT bufferSize[ ], UINT bytesValid)

Fill AAC decoder’s internal input buffer with bitstream data from the external input buffer. The function  only copies such data as long as the decoder-internal input buffer is not full. So it grabs whatever it can  from pBuffer and returns information (bytesValid) so that at a subsequent call of aacDecoder_Fill(), the  right position in pBuffer can be determined to grab the next data.

· LINKSPEC_H AAC_DECODER_ERROR  aacDecoder_DecodeFrame (HANDLE_-AACDECODER self, INT_PCM                                     pTimeData, const INT timeDataSize, const UINT flags)

decode one audio frame.

· LINKSPEC_H void  aacDecoder_Close (HANDLE_AACDECODER self)

De-allocate all resources of an AAC decoder instance.

· LINKSPEC_H CStreamInfo  aacDecoder_GetStreamInfo (HANDLE_AACDECODER self)

Get CStreamInfo handle from decoder.

· LINKSPEC_H INT  aacDecoder_GetLibInfo (LIB_INFO info)

Get decoder library info

4：function call sequence

   4.1. Call aacDecoder_Open() to open and retrieve a handle to a new AAC decoder instance.

       aacDecoderInfo = aacDecoder_Open(mpegFileRead_GetTransportType(hDataSrc), nrOfLayers);

   4.2  If out-of-band config data (Audio Specific Config (ASC) or Stream Mux Config (SMC)) is available,  call aacDecoder_ConfigRaw() to pass it to the decoder and before the decoding process starts. If this  data is not available in advance, the decoder will get it from the bitstream and configure itself while  decoding with aacDecoder_DecodeFrame().

   4.3. Begin decoding loop.   do {

   4.4  Read data from bitstream file or stream into a client-supplied input buffer("inBuffer" in main.cpp).  If it is very small like just 4, aacDecoder_DecodeFrame() will repeatedly return AAC_DEC_NOT_-ENOUGH_BITS until enough bits were fed by aacDecoder_Fill(). Only read data when this buffer  has completely been processed and is then empty. For file-based input execute mpegFileRead_Read()  or any other implementation with similar functionality.

   4.5  Call aacDecoder_Fill() to fill the decoder’s internal bitstream input buffer with the client-suppliedexternal bitstream input buffer.

          aacDecoder_Fill(aacDecoderInfo, inBuffer, bytesRead, bytesValid);

   4.6   Call aacDecoder_DecodeFrame() which writes decoded PCM audio data to a client-supplied buffer.It is the client’s responsibility to allocate a buffer which is large enough to hold this output data.

          ErrorStatus = aacDecoder_DecodeFrame(aacDecoderInfo, TimeData, OUT_BUF_SIZE, flags);

     If the bitstream’s configuration (number of channels, sample rate, frame size) is not known in ad-vance, you may call aacDecoder_GetStreamInfo() to retrieve a structure containing this information  and then initialize an audio output device. In the example main.cpp, if the number of channels or  the sample rate has changed since program start or since the previously decoded frame, the audio  output device will be re-initialized. If WAVE file output is chosen, a new WAVE file for each new  configuration will be created.

   4.7  Repeat steps 5 to 7 until no data to decode is available anymore, or if an error occured.

    

   4.8  Call aacDecoder_Close() to de-allocate all AAC decoder and transport layer structures.

5：buffer system

There are three main buffers in an AAC decoder application. One external input buffer to hold bitstream  data from file I/O or elsewhere, one decoder-internal input buffer, and one to hold the decoded output PCM  sample data, whereas this output buffer may overlap with the external input buffer.

The external input buffer is set in the example framework main.cpp and its size is defined by IN_BUF_-SIZE. You may freely choose different sizes here. To feed the data to the decoder-internal input buffer, use  the function aacDecoder_Fill(). This function returns important information about how many bytes in the  external input buffer have not yet been copied into the internal input buffer (variable bytesValid). Once the  external buffer has been fully copied, it can be re-filled again. In case you want to re-fill it when there are

still unprocessed bytes (bytesValid is unequal 0), you would have to additionally perform a memcpy(), so  that just means unnecessary computational overhead and therefore we recommend to re-fill the buffer only  when bytesValid is 0.