Java Jsoup - FIX for exception com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input in field name at [Source: java.io.StringReader@1725dc0f; line: 1, column: 1046939]

1. Problem description

Code reproduction to get exception, json from my spring controller backend is quite big about 30 MB

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.jsoup.Connection;
import org.jsoup.Jsoup;

import java.io.IOException;

public class JsoupHttpGetApp {

    public static void main(String[] args) throws IOException {

        String jsonBodyStr;
        try {
            jsonBodyStr = Jsoup.connect("http://localhost:8080/get-all-users")
                    .followRedirects(true)
                    .ignoreHttpErrors(true)
                    .userAgent("Mozilla/5.0 AppleWebKit/537.36 (KHTML," +
                            " like Gecko) Chrome/45.0.2454.4 Safari/537.36")
                    .method(Connection.Method.GET)
                    .timeout(10_000)
                    .ignoreContentType(true)
                    .execute()
                    .body();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

        JsonNode rootNode = new ObjectMapper(new JsonFactory())
                .readTree(jsonBodyStr);

        System.out.println(rootNode);
    }
}

With above code I get this exception:

Exception in thread "main" com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing quote for a string value
 at [Source: java.io.StringReader@1a0dcaa; line: 1, column: 1046947]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:483)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2039)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2026)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:276)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:239)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:273)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:72)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:16)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3789)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2372)

2. Solution

As we can see from stack trace the default size of json response in Jsoup is 1046947 bytes (1_046_947).

As fix for this we need to set bigger max body size.

Jsoup.connect("http://localhost:8080/get-all-users")
    // FIX - this line set max response size without limits
    // the only limit is your machine 
	.maxBodySize(0)
    // ...

From Jsoup.maxBodySize(0) java docs:

Set the maximum bytes to read from the (uncompressed) connection into the body, before the connection is closed, and the input truncated.

The default maximum is 1MB.

A max size of zero is treated as an infinite amount (bounded only by your patience and the memory available on your machine).

@param bytes number of bytes to read from the input before truncating
@return this Connection, for chaining

OR

Jsoup.connect("http://localhost:8080/get-all-users")
    // FIX - this line set max response size to 100 MB
	.maxBodySize(1_000_000 * 100) // 100 mb ~
    // ...

Full working code: 

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.jsoup.Connection;
import org.jsoup.Jsoup;

import java.io.IOException;

public class JsoupHttpGetAppFIX {

    public static void main(String[] args) throws IOException {

        String jsonBodyStr;
        try {
            jsonBodyStr = Jsoup.connect("http://localhost:8080/get-all-users")

                    // FIX - this line set max response size without limit
                    // all our data will be fetched
                    .maxBodySize(0)

                    .followRedirects(true)
                    .ignoreHttpErrors(true)
                    .userAgent("Mozilla/5.0 AppleWebKit/537.36 (KHTML," +
                            " like Gecko) Chrome/45.0.2454.4 Safari/537.36")
                    .method(Connection.Method.GET)
                    .timeout(10_000)
                    .ignoreContentType(true)
                    .execute()
                    .body();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

        JsonNode rootNode = new ObjectMapper(new JsonFactory())
                .readTree(jsonBodyStr);

        System.out.println(rootNode);
    }
}