An ffmpeg and SDL Tutorial

An ffmpeg and SDL TutorialPage 1 2 3 4 5 6 7 8 End Prev Home NextTutorial 01: Making ScreencapsCode: tutorial01.cOverviewMovie files have a few basic components. First, the file itself is called a container, and the type of container determines where the information in the file goes. Examples of containers are AVI and Quicktime. Next, you have a bunch of streams; for example, you usually have an audio stream and a video stream. (A "stream" is just a fancy word for "a succession of data elements made available over time".) The data elements in a stream are called frames. Each stream is encoded by a different kind of codec. The codec defines how the actual data is COded and DECoded - hence the name CODEC. Examples of codecs are DivX and MP3. Packets are then read from the stream. Packets are pieces of data that can contain bits of data that are decoded into raw frames that we can finally manipulate for our application. For our purposes, each packet contains complete frames, or multiple frames in the case of audio. At its very basic level, dealing with video and audio streams is very easy:10 OPEN video_stream FROM video.avi 20 READ packet FROM video_stream INTO frame 30 IF frame NOT COMPLETE GOTO 20 40 DO SOMETHING WITH frame 50 GOTO 20 Handling multimedia with ffmpeg is pretty much as simple as this program, although some programs might have a very complex "DO SOMETHING" step. So in this tutorial, we're going to open a file, read from the video stream inside it, and our DO SOMETHING is going to be writing the frame to a PPM file.Opening the FileFirst, let's see how we open a file in the first place. With ffmpeg, you have to first initialize the library. (Note that some systems might have to use <ffmpeg/avcodec.h> and <ffmpeg/avformat.h> instead.)#include <avcodec.h> #include <avformat.h> ...int main(int argc, charg *argv[]) { av_register_all(); This registers all available file formats and codecs with the library so they will be used automatically when a file with the corresponding format/codec is opened. Note that you only need to call av_register_all() once, so we do it here in main(). If you like, it's possible to register only certain individual file formats and codecs, but there's usually no reason why you would have to do that. Now we can actually open the file:AVFormatContext *pFormatCtx; // Open video file if(av_open_input_file(&pFormatCtx, argv[1], NULL, 0, NULL)!=0) return -1; // Couldn't open file We get our filename from the first argument. This function reads the file header and stores information about the file format in the AVFormatContext structure we have given it. The last three arguments are used to specify the file format, buffer size, and format options, but by setting this to NULL or 0, libavformat will auto-detect these. This function only looks at the header, so next we need to check out the stream information in the file.:// Retrieve stream information if(av_find_stream_info(pFormatCtx)<0) return -1; // Couldn't find stream information This function populates pFormatCtx->streams with the proper information. We introduce a handy debugging function to show us what's inside: // Dump information about file onto standard error dump_format(pFormatCtx, 0, argv[1], 0); Now pFormatCtx->streams is just an array of pointers, of sizepFormatCtx->nb_streams, so let's walk through it until we find a video stream.int i; AVCodecContext *pCodecCtx;// Find the first video stream videoStream=-1; for(i=0; i<pFormatCtx->nb_streams; i++) if(pFormatCtx->streams[i]->codec->codec_type==CODEC_TYPE_VIDEO) { videoStream=i; break; }if(videoStream==-1) return -1; // Didn't find a video stream// Get a pointer to the codec context for the video stream pCodecCtx=pFormatCtx->streams[videoStream]->codec; The stream's information about the codec is in what we call the "codec context." This contains all the information about the codec that the stream is using, and now we have a pointer to it. But we still have to find the actual codec and open it: AVCodec *pCodec;// Find the decoder for the video stream pCodec=avcodec_find_decoder(pCodecCtx->codec_id); if(pCodec==NULL) { fprintf(stderr, "Unsupported codec!\n"); return -1; // Codec not found } // Open codec if(avcodec_open(pCodecCtx, pCodec)<0) return -1; // Could not open codec Some of you might remember from the old tutorial that there were two other parts to this code: adding CODEC_FLAG_TRUNCATED to pCodecCtx->flags and adding a hack to correct grossly incorrect frame rates. These two fixes aren't in ffplay.c anymore, so I have to assume that they are not necessary anymore. There's another difference to point out since we removed that code: pCodecCtx->time_base now holds the frame rate information.time_base is a struct that has the numerator and denominator (AVRational). We represent theframe rate as a fraction because many codecs have non-integer frame rates (like NTSC's 29.97fps).Storing the DataNow we need a place to actually store the frame:AVFrame *pFrame; // Allocate video frame pFrame=avcodec_alloc_frame(); Since we're planning to output PPM files, which are stored in 24-bit RGB, we're going to have to convert our frame from its native format to RGB. ffmpeg will do these conversions for us. For most projects (including ours) we're going to want to convert our initial frame to a specific format. Let's allocate a frame for the converted frame now. // Allocate an AVFrame structure pFrameRGB=avcodec_alloc_frame();if(pFrameRGB==NULL) return -1; Even though we've allocated the frame, we still need a place to put the raw data when we convert it. We use avpicture_get_size to get the size we need, and allocate the space manually: uint8_t *buffer; int numBytes; // Determine required buffer size and allocate buffer numBytes=avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height); buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));av_malloc is ffmpeg's malloc that is just a simple wrapper around malloc that makes sure thememory addresses are aligned and such. It will not protect you from memory leaks, double freeing, or other malloc problems. Now we use avpicture_fill to associate the frame with our newly allocated buffer. About the AVPicture cast: the AVPicture struct is a subset of the AVFrame struct - the beginning of the AVFrame struct is identical to the AVPicture struct.// Assign appropriate parts of buffer to image planes in pFrameRGB // Note that pFrameRGB is an AVFrame, but AVFrame is a superset // of AVPicture avpicture_fill((AVPicture *)pFrameRGB, buffer, PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height); Finally! Now we're ready to read from the stream!Reading the DataWhat we're going to do is read through the entire video stream by reading in the packet, decoding it into our frame, and once our frame is complete, we will convert and save it.int frameFinished; AVPacket packet;i=0; while(av_read_frame(pFormatCtx, &packet)>=0) { // Is this a packet from the video stream? if(packet.stream_index==videoStream) { // Decode video frame avcodec_decode_video(pCodecCtx, pFrame, &frameFinished, packet.data, packet.size);// Did we get a video frame?if(frameFinished) { // Convert the image from its native format to RGB img_convert((AVPicture *)pFrameRGB, PIX_FMT_RGB24, (AVPicture*)pFrame, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);// Save the frame to disk if(++i<=5) SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height, i); } }// Free the packet that was allocated by av_read_frame av_free_packet(&packet); }A note on packetsTechnically a packet can contain partial frames or other bits of data, but ffmpeg's parser ensures that the packets we get contain either complete or multiple frames.The process, again, is simple: av_read_frame() reads in a packet and stores it in theAVPacket struct. Note that we've only allocated the packet structure - ffmpeg allocates the internal data for us, which is pointed to by packet.data. This is freed by the av_free_packet() later. avcodec_decode_video() converts the packet to aframe for us. However, we might not have all the information we need for a frame after decoding a packet, so avcodec_decode_video() sets frameFinished for us when we have the next frame. Finally, we use img_convert() to convert from the native format (pCodecCtx->pix_fmt) to RGB. Remember that you can cast an AVFrame pointer to an AVPicture pointer. Finally, we pass the frame and height and width information to our SaveFrame function. Now all we need to do is make the SaveFrame function to write the RGB information to a file in PPM format. We're going to be kind of sketchy on the PPM format itself; trust us, it works.void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame) { FILE *pFile; char szFilename[32]; int y;// Open filesprintf(szFilename, "frame%d.ppm", iFrame); pFile=fopen(szFilename, "wb"); if(pFile==NULL) return;// Write header fprintf(pFile, "P6\n%d %d\n255\n", width, height);// Write pixel data for(y=0; y<height; y++) fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);// Close file fclose(pFile); } We do a bit of standard file opening, etc., and then write the RGB data. We write the file one line at a time. A PPM file is simply a file that has RGB information laid out in a long string. If you know HTML colors, it would be like laying out the color of each pixel end to end like#ff0000#ff0000.... would be a red screen. (It's stored in binary and without the separator,but you get the idea.) The header indicated how wide and tall the image is, and the max size of the RGB values. Now, going back to our main() function. Once we're done reading from the video stream, we just have to clean everything up:// Free the RGB image av_free(buffer); av_free(pFrameRGB);// Free the YUV frame av_free(pFrame);// Close the codec avcodec_close(pCodecCtx);// Close the video file av_close_input_file(pFormatCtx);return 0; You'll notice we use av_free for the memory we allocated with avcode_alloc_frame and av_malloc. That's it for the code! Now, if you're on Linux or a similar platform, you'll run:gcc -o tutorial01 tutorial01.c -lavutil -lavformat -lavcodec -lz -lavutil -lm If you have an older version of ffmpeg, you may need to drop -lavutil: gcc -o tutorial01 tutorial01.c -lavformat -lavcodec -lz -lm Most image programs should be able to open PPM files. Test it on some movie files.>> Tutorial 2: Outputting to the ScreenTutorial 02: Outputting to the ScreenCode: tutorial02.cSDL and VideoTo draw to the screen, we're going to use SDL. SDL stands for Simple Direct Layer, and is an excellent library for multimedia, is cross-platform, and is used in several projects. You can get the library at the official website or you can download the development package for your operating system if there is one. You'll need the libraries to compile the code for this tutorial (and for the rest of them, too). SDL has many methods for drawing images to the screen, and it has one in particular that is meant for displaying movies on the screen - what it calls a YUV overlay. YUV (technically not YUV but YCbCr) * A note: There is a great deal of annoyance from some people at the convention of calling "YCbCr" "YUV". Generally speaking, YUV is an analog format and YCbCr is a digital format. ffmpeg and SDL both refer to YCbCr as YUV in their code and macros. is a way of storing raw image data like RGB. Roughly speaking, Y is the brightness (or "luma") component, and U and V are the color components. (It's more complicated than RGB because some of the color information is discarded, and you might have only 1 U and V sample for every 2 Y samples.) SDL's YUV overlay takes in a raw array of YUV data and displays it. It accepts 4 different kinds of YUV formats, but YV12 is the fastest. There is another YUV format called YUV420P that is the same as YV12, except the U and V arrays are switched. The 420 means it is subsampled at a ratio of 4:2:0, basically meaning there is 1 color sample for every 4 luma samples, so the color information is quartered. This is a good way of saving bandwidth, as the human eye does not percieve this change. The "P" in the name means that the format is "planar" — simply meaning that the Y, U, and V components are in separate arrays. ffmpeg can convert images to YUV420P, with the added bonus that many video streams are in that format already, or are easily converted to that format. So our current plan is to replace the SaveFrame() function from Tutorial 1, and instead output our frame to the screen. But first we have to start by seeing how to use the SDL Library. First we have to include the libraries and initalize SDL:#include <SDL.h>#include <SDL_thread.h> if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) { fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError()); exit(1); }SDL_Init() essentially tells the library what features we're going to use. SDL_GetError(), of course, is a handy debugging function.Creating a DisplayNow we need a place on the screen to put stuff. The basic area for displaying images with SDL is called a surface:SDL_Surface *screen;screen = SDL_SetVideoMode(pCodecCtx->width, pCodecCtx->height, 0, 0); if(!screen) { fprintf(stderr, "SDL: could not set video mode - exiting\n"); exit(1); } This sets up a screen with the given width and height. The next option is the bit depth of the screen - 0 is a special value that means "same as the current display". (This does not work on OS X; see source.) Now we create a YUV overlay on that screen so we can input video to it:SDL_Overlay*bmp;bmp = SDL_CreateYUVOverlay(pCodecCtx->width, pCodecCtx->height, SDL_YV12_OVERLAY, screen); As we said before, we are using YV12 to display the image.Displaying the ImageWell that was simple enough! Now we just need to display the image. Let's go all the way down to where we had our finished frame. We can get rid of all that stuff we had for the RGB frame, and we're going to replace the SaveFrame() with our display code. To display the image, we're going to make an AVPicture struct and set its data pointers and linesize to our YUV overlay:if(frameFinished) { SDL_LockYUVOverlay(bmp);AVPicture pict; pict.data[0] = bmp->pixels[0]; pict.data[1] = bmp->pixels[2]; pict.data[2] = bmp->pixels[1];pict.linesize[0] = bmp->pitches[0]; pict.linesize[1] = bmp->pitches[2]; pict.linesize[2] = bmp->pitches[1];// Convert the image into YUV format that SDL uses img_convert(&pict, PIX_FMT_YUV420P, (AVPicture *)pFrame, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);SDL_UnlockYUVOverlay(bmp); } First, we lock the overlay because we are going to be writing to it. This is a good habit to get into so you don't have problems later. The AVPicture struct, as shown before, has a data pointer that is an array of 4 pointers. Since we are dealing with YUV420P here, we only have 3 channels, and therefore only 3 sets of data. Other formats might have a fourth pointer for an alpha channel or something. linesize is what it sounds like. The analogous structures in our YUV overlay are the pixels and pitches variables. ("pitches" is the term SDL uses to refer to the width of a given line of data.) So what we do is point the three arrays of pict.data at our overlay, so when we write to pict, we're actually writing into our overlay, which of course already has the necessary space allocated. Similarly, we get the linesize information directly from our overlay. We change the conversion format to PIX_FMT_YUV420P, and we use img_convert just like before.Drawing the ImageBut we still need to tell SDL to actually show the data we've given it. We also pass this function a rectangle that says where the movie should go and what width and height it should be scaled to. This way, SDL does the scaling for us, and it can be assisted by your graphics processor for faster scaling:SDL_Rect rect;if(frameFinished) { /* ... code ... */ // Convert the image into YUV format that SDL uses img_convert(&pict, PIX_FMT_YUV420P, (AVPicture *)pFrame, pCodecCtx->pix_fmt,pCodecCtx->width, pCodecCtx->height); SDL_UnlockYUVOverlay(bmp); rect.x = 0; rect.y = 0; rect.w = pCodecCtx->width; rect.h = pCodecCtx->height; SDL_DisplayYUVOverlay(bmp, &rect); } Now our video is displayed! Let's take this time to show you another feature of SDL: its event system. SDL is set up so that when you type, or move the mouse in the SDL application, or send it a signal, it generates an event. Your program then checks for these events if it wants to handle user input. Your program can also make up events to send the SDL event system. This is especially useful when multithread programming with SDL, which we'll see in Tutorial 4. In our program, we're going to poll for events right after we finish processing a packet. For now, we're just going to handle the SDL_QUIT event so we can exit:SDL_Eventevent;av_free_packet(&packet); SDL_PollEvent(&event); switch(event.type) { case SDL_QUIT: SDL_Quit(); exit(0); break; default: break; } And there we go! Get rid of all the old cruft, and you're ready to compile. If you are using Linux or a variant, the best way to compile using the SDL libs is this: gcc -o tutorial02 tutorial02.c -lavutil -lavformat -lavcodec -lz -lm \ `sdl-config --cflags --libs` sdl-config just prints out the proper flags for gcc to include the SDL libraries properly. You may need to do something different to get it to compile on your system; please check the SDL documentation for your system. Once it compiles, go ahead and run it. What happens when you run this program? The video is going crazy! In fact, we're just displaying all the video frames as fast as we can extract them from the movie file. We don't have any coderight now for figuring out when we need to display video. Eventually (in Tutorial 5), we'll get around to syncing the video. But first we're missing something even more important: sound!>> Playing SoundTutorial 03: Playing SoundCode: tutorial03.cAudioSo now we want to play sound. SDL also gives us methods for outputting sound. TheSDL_OpenAudio() function is used to open the audio device itself. It takes as arguments an SDL_AudioSpec struct, which contains all the information about the audio we are going tooutput. Before we show how you set this up, let's explain first about how audio is handled by computers. Digital audio consists of a long stream of samples. Each sample represents a value of the audio waveform. Sounds are recorded at a certain sample rate, which simply says how fast to play each sample, and is measured in number of samples per second. Example sample rates are 22,050 and 44,100 samples per second, which are the rates used for radio and CD respectively. In addition, most audio can have more than one channel for stereo or surround, so for example, if the sample is in stereo, the samples will come 2 at a time. When we get data from a movie file, we don't know how many samples we will get, but ffmpeg will not give us partial samples - that also means that it will not split a stereo sample up, either. SDL's method for playing audio is this: you set up your audio options: the sample rate (called "freq" for frequency in the SDL struct), number of channels, and so forth, and we also set a callback function and userdata. When we begin playing audio, SDL will continually call this callback function and ask it to fill the audio buffer with a certain number of bytes. After we put this information in theSDL_AudioSpec struct, we call SDL_OpenAudio(), which will open the audio deviceand give us back another AudioSpec struct. These are the specs we will actually be using — we are not guaranteed to get what we asked for!Setting Up the AudioKeep that all in your head for the moment, because we don't actually have any information yet about the audio streams yet! Let's go back to the place in our code where we found the video stream and find which stream is the audio stream.// Find the first video stream videoStream=-1; audioStream=-1; for(i=0; i < pFormatCtx->nb_streams; i++) {if(pFormatCtx->streams[i]->codec->codec_type==CODEC_TYPE_VIDEO && videoStream < 0) { videoStream=i; } if(pFormatCtx->streams[i]->codec->codec_type==CODEC_TYPE_AUDIO && audioStream < 0) { audioStream=i; } } if(videoStream==-1) return -1; // Didn't find a video stream if(audioStream==-1) return -1; From here we can get all the info we want from the AVCodecContext from the stream, just like we did with the video stream: AVCodecContext *aCodecCtx;aCodecCtx=pFormatCtx->streams[audioStream]->codec;Contained within this codec context is all the information we need to set up our audio:wanted_spec.freq = aCodecCtx->sample_rate; wanted_spec.format = AUDIO_S16SYS; wanted_spec.channels = aCodecCtx->channels; wanted_spec.silence = 0; wanted_spec.samples = SDL_AUDIO_BUFFER_SIZE; wanted_spec.callback = audio_callback; wanted_erdata = aCodecCtx;if(SDL_OpenAudio(&wanted_spec, &spec) < 0) { fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError()); return -1; } Let's go through these:• •freq: The sample rate, as explained earlier. format: This tells SDL what format we will be giving it. The "S" in "S16SYS" stands for"signed", the 16 says that each sample is 16 bits long, and "SYS" means that the endian-order will depend on the system you are on. This is the format that•avcodec_decode_audio2 will give us the audio in. channels: Number of audio channels.• • • •silence: This is the value that indicated silence. Since the audio is signed, 0 is ofcourse the usual value.samples: This is the size of the audio buffer that we would like SDL to give us when itasks for more audio. A good value here is between 512 and 8192; ffplay uses 1024.callback: Here's where we pass the actual callback function. We'll talk more about thecallback function later.userdata: SDL will give our callback a void pointer to any user data that we want ourcallback function to have. We want to let it know about our codec context; you'll see why.Finally, we open the audio with SDL_OpenAudio. If you remember from the previous tutorials, we still need to open the audio codec itself. This is straightforward:AVCodec*aCodec;aCodec = avcodec_find_decoder(aCodecCtx->codec_id); if(!aCodec) { fprintf(stderr, "Unsupported codec!\n"); return -1; } avcodec_open(aCodecCtx, aCodec);QueuesThere! Now we're ready to start pulling audio information from the stream. But what do we do with that information? We are going to be continuously getting packets from the movie file, but at the same time SDL is going to call the callback function! The solution is going to be to create some kind of global structure that we can stuff audio packets in so our audio_callback has something to get audio data from! So what we're going to do is to create a queue of packets. ffmpeg even comes with a structure to help us with this: AVPacketList, which is just a linked list for packets. Here's our queue structure:typedef struct PacketQueue { AVPacketList *first_pkt, *last_pkt; int nb_packets; int size; SDL_mutex *mutex; SDL_cond *cond; } PacketQueue; First, we should point out that nb_packets is not the same as size — size refers to a byte size that we get from packet->size. You'll notice that we have a mutex and a condtion variable in there. This is because SDL is running the audio process as a separate thread. If we don'tlock the queue properly, we could really mess up our data. We'll see how in the implementation of the queue. Every programmer should know how to make a queue, but we're including this so you can learn the SDL functions. First we make a function to initialize the queue:void packet_queue_init(PacketQueue *q) { memset(q, 0, sizeof(PacketQueue)); q->mutex = SDL_CreateMutex(); q->cond = SDL_CreateCond(); } Then we will make a function to put stuff in our queue: int packet_queue_put(PacketQueue *q, AVPacket *pkt) { AVPacketList *pkt1; if(av_dup_packet(pkt) < 0) { return -1; } pkt1 = av_malloc(sizeof(AVPacketList)); if (!pkt1) return -1; pkt1->pkt = *pkt; pkt1->next = NULL;SDL_LockMutex(q->mutex);if (!q->last_pkt) q->first_pkt = pkt1; else q->last_pkt->next = pkt1; q->last_pkt = pkt1; q->nb_packets++; q->size += pkt1->pkt.size; SDL_CondSignal(q->cond);SDL_UnlockMutex(q->mutex); return 0; }SDL_LockMutex() locks the mutex in the queue so we can add something to it, and then SDL_CondSignal() sends a signal to our get function (if it is waiting) through our conditionvariable to tell it that there is data and it can proceed, then unlocks the mutex to let it go.Here's the corresponding get function. Notice how SDL_CondWait() makes the function block (i.e. pause until we get data) if we tell it to.int quit = 0; static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block) { AVPacketList *pkt1; int ret;SDL_LockMutex(q->mutex);for(;;) {if(quit) { ret = -1; break; }pkt1 = q->first_pkt; if (pkt1) { q->first_pkt = pkt1->next; if (!q->first_pkt) q->last_pkt = NULL; q->nb_packets--; q->size -= pkt1->pkt.size; *pkt = pkt1->pkt; av_free(pkt1); ret = 1; break; } else if (!block) { ret = 0; break; } else { SDL_CondWait(q->cond, q->mutex); } } SDL_UnlockMutex(q->mutex); return ret; } As you can see, we've wrapped the function in a forever loop so we will be sure to get some data if we want to block. We avoid looping forever by making use of SDL's SDL_CondWait() function. Basically, all CondWait does is wait for a signal from SDL_CondSignal() (orSDL_CondBroadcast()) and then continue. However, it looks as though we've trapped itwithin our mutex — if we hold the lock, our put function can't put anything in the queue! However,what SDL_CondWait() also does for us is to unlock the mutex we give it and then attempt to lock it again once we get the signal.In Case of FireYou'll also notice that we have a global quit variable that we check to make sure that we haven't set the program a quit signal (SDL automatically handles TERM signals and the like). Otherwise, the thread will continue forever and we'll have to kill-9 the program. ffmpeg also has afunction for providing a callback to check and see if we need to quit some blocking function:url_set_interrupt_cb.int decode_interrupt_cb(void) { return quit; } ... main() { ... url_set_interrupt_cb(decode_interrupt_cb); ... SDL_PollEvent(&event); switch(event.type) { case SDL_QUIT: quit = 1; ... This only applies for ffmpeg functions that block, of course, not SDL ones. We make sure to set thequit flag to 1.Feeding PacketsThe only thing left is to set up our queue:PacketQueue audioq; main() { ... avcodec_open(aCodecCtx, aCodec);packet_queue_init(&audioq); SDL_PauseAudio(0);SDL_PauseAudio() finally starts the audio device. It plays silence if it doesn't get data;which it won't right away. So, we've got our queue set up, now we're ready to start feeding it packets. We go to our packet-reading loop:。