Steam.CopyTo - Performance problems

This week I heard about an interesting performance problem, that I want to share with you.

There is a desktop application that is used to import data from files to database. These files were in different formats, from XML to binary. Everything was fine, the application work great in the development and first phase of testing. All the code was double check, each stream was dispose when it was no longer used.

During the testing phase, the application started to have an odd behavior, the RAM memory started to go crazy. Normally, the application use around 200MB during an import, but there were times when more than 1500MB of ram were consumed. Not only this, but “Out of memory” exception appeared a lot of time during different imports. All this started too appeared at the testing phase.

With the same setup, developers manage to reproduce the problem and using the memory profiles that comes with Visual Studio the problem was spotted.

Stream stream1 = …
Stream stream2 = …
stream2.CopyTo(stream1);

The problem was with CopyTo method. Because this method was called a lot of time in a short period of time, the GC didn’t had time to dispose all the resources used during the copy process. This can become a big problem when the size of stream is pretty big. This method was introduce in .NET 4.0 and the scope of it is to copy the content of one stream to another. This is done by reading to a buffer of bytes (default buffer size is 4096) chunks from the stream and write them to the second stream. 

It is very similar with what we usually do in .NET 3.5 when we had to copy one stream to another:

public void CopyStream(Stream input, Stream output)
{
    byte[] buffer = new byte[size];
    int read;
    while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write (buffer, 0, read);
    }
}

The problem is with the buffer that is represented as an array of bytes. If we call the CopyTo method from time to time with small streams will not have any kind of problem. But when we use big streams intensive, calling this method very often can create huge problems. Each time when we make new calls to this method, a new part of our memory will be allocated for the buffer. GC will not have time to clean the memory and we can end up with a lot of memory that is not disposed.

The solution for this problem was pretty simple. The CopyTo method was replaced with a BinaryReader that was used to copy the content of stream to an array of bytes. This array of bytes was later used to create the MemoryStream. In this way, the stream that was used for read data is disposed immediately (BinaryReader) and the array of bytes that is used as a buffer is used directly in the memory stream.

byte[] bytes;
using( BinaryReader reader = new BinaryReader(inputStream))
{
    bytes = reader.ReadBytes((int)inputStream.Length); 
}
outputStream = new MemoryStream(bytes, 0, bytes.Count());

Using this solution, the problem with the memory leak disappeared. It seems that not all time, the new API is the best solution when we need performance.