Generating large ZIP files

Lets imagine a Java application which receives a set of files encoded with Base64 from one system, generates a ZIP archive with all those files, and then pushes it as a Base64 encoded form into another system.

Sounds easy. Lets take each file as a String, decode it from Base64 to a byte array. Then create a ZipOutputStream connected to a ByteArrayOutputStream, write all the files to the ZIP stream. Then take a ZIP data as a byte array, encoded it with Base64 into a String and send it out. Job done.

But what happens if you receive 10 files, each 4MB large. You have 40MB of objects in memory. Not a big deal for a modern JVM. But then you decode each of them into a byte array so you get another 10 objects, 3MB large each (additional 30MB) then you ZIP them. For files like PDF or JPG even high compression does not make the ZIP smaller so you end up with approximately 30MB ZIP file. And additional 40MB after Base64 encoding. It means dozens of large object allocations.

The problem gets even worse if you take a look at the standard Java classes used here. ByteArrayOutputStream encapsulates a byte array which can expand when needed. By default it creates 32 bytes. If you’re trying to write more it doubles its size by creating a new 64 byte array and copying first 32 bytes from the old one. If it still cannot fit the data it creates 128 byte array, copies first 64 bytes …… and does it over and over again until the array fits your 30MB archive!

There is even more! In order to get the raw data you just zipped you have to call ByteArrayOutputStream.toByteArray() which … returns a copy of an internal byte array (allocates new array and copies the internal array into it).

While dealing with Base64 you would probably need to create a String from a byte array or get bytes from a String and guess what: both operations create another copies as String are immutable.

I hope you’re still following and keeping track of all the objects created so far because I am already lost in my calculations ;) As a result a simple operation may render an OutOfMemoryError even for a relatively small files to be archived.

So what is wrong with the above implementation? It keeps everything in memory. A solution of the above problem is to process streams of bytes on the fly and avoid keeping the data in memory, avoid copying data when it is not needed.

I made a simple exercise and wrote a small piece of code which compresses a set of files into a ZIP files. I have successfully compressed 5 files of 200MB each within a 4MB of heap!

Yep, -Xmx4m was more then enough.

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
 
public class BigZip {
 
    byte buffer[] = new byte[8192];
 
    public static void main(String[] args) throws Exception {
        new BigZip().compress(args);
    }
 
    private void put(ZipOutputStream zos, File file) throws Exception {
        zos.putNextEntry(new ZipEntry(file.getName()));
        FileInputStream fin = new FileInputStream(file);
        int read = 0;
        while ((read = fin.read(buffer)) > 0) {
            zos.write(buffer, 0, read);
        }
        fin.close();
        zos.closeEntry();
        zos.flush();
    }
 
    private void compress(String[] files) throws Exception {
        FileOutputStream os = new FileOutputStream("result.zip");
        ZipOutputStream zos = new ZipOutputStream(os);
        for (String f: files) {
            put(zos, new File(f));
        }
        zos.flush();
        zos.close();
    }
 
}