How to handle large files in Java
A file becomes "large" in Java at three points: when it no longer fits in the heap, when it passes the 2 GB limit of arrays and buffers, and when reading it takes long enough to matter. This page shows each limit on real files, measures what a loaded line costs in memory, where the time of a big job really goes, and how to split a file across threads.
Contents
What makes a file large in Java Loading versus streaming a file bigger than the heap What a loaded line costs on the heap Where the time goes: reading versus parsing Buffer size for byte streams Files over 2 GB: arrays, mapping windows and positional reads Processing byte ranges in parallel Choosing a strategy
What makes a file large in Java
Size alone does not decide whether a file is a problem. What matters is how it compares with three limits:
- The heap: loading a file creates Java objects, and they take several times more memory than the bytes on disk. A 100 MB CSV file can need 300 MB of heap once it is a list of strings.
- The 2 GB index limit: arrays,
,String andByteBuffer are indexed with anMappedByteBuffer , so none of them can hold more than 2,147,483,647 elements. A 3 GB file cannot be read into oneint or mapped as one buffer, however much memory the machine has.byte[] - Time: reading itself is fast, often several GB per second from the operating system's cache. In a slow job the time usually goes into what the program does with each line, not into the reading.
The strategies for large files follow from these limits. Most avoid holding the file at all; the rest work around the 2 GB limit or spread the work over several cores.
| Strategy | Since | Memory | Best for |
|---|---|---|---|
Stream lines: BufferedReaderFiles.lines | Java 1.1, Java 8 | One line | Line-based text of any size |
Stream bytes with a large buffer: InputStream.read(byte[]) | Java 1.0 | One buffer | Binary files, copying, checksums |
Positional reads: FileChannel.read(buf, position) | Java 1.4 | One buffer | Any part of a file of any size |
Memory mapping in windows: FileChannel.map | Java 1.4 | Outside the heap, paged in by the OS | Byte-level scanning and random access |
One mapping of any size: FileChannel.map(..., Arena) | Java 22 (JEP 454) | Outside the heap | Files over 2 GB without windows |
| Byte ranges processed in parallel | Any | One buffer per thread | CPU-heavy parsing on many cores |
The basic reading APIs and their relative speed on an ordinary file are covered on
Loading versus streaming a file bigger than the heap
The first program creates a CSV file with 4 million rows, about 108 MB, and runs with the heap limited to 64 MB (-Xmx64mFiles.readAllLinesBufferedReader
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
public class LargeFileMemory {
// Writes a header and rows like "123,product-123,1.23"
static Path createCsv(int rows) throws IOException {
Path file = Files.createTempFile("large", ".csv");
try (BufferedWriter w = Files.newBufferedWriter(file)) {
w.write("id,name,price\n");
for (int i = 1; i <= rows; i++) {
w.write(i + ",product-" + i + "," + price(i) + "\n");
}
}
return file;
}
public static void main(String[] args) throws IOException {
long maxHeapMb = Runtime.getRuntime().maxMemory() / (1024 * 1024);
Path csv = createCsv(4_000_000);
System.out.printf("max heap: %d MB, file size: %d MB%n", maxHeapMb,
Files.size(csv) / (1024 * 1024));
// Loading every line at once needs far more memory than the file size
try {
List<String> all = Files.readAllLines(csv);
System.out.println("readAllLines: loaded " + all.size() + " lines");
} catch (OutOfMemoryError e) {
System.out.println("readAllLines: OutOfMemoryError (" + e.getMessage() + ")");
}
// Streaming keeps only the current line in memory
long rows = 0;
double total = 0;
try (BufferedReader r = Files.newBufferedReader(csv)) {
r.readLine(); // skip the header
String line;
while ((line = r.readLine()) != null) {
total += Double.parseDouble(line.substring(line.lastIndexOf(',') + 1));
rows++;
}
}
System.out.printf("streamed %,d rows, total price %,.2f%n", rows, total);
Files.delete(csv);
}
// 0.00, 0.01, ... 9.99, then again from 0.00: always two decimals
static String price(int i) {
int cents = i % 1000;
return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
}
}
Output:
max heap: 64 MB, file size: 108 MB
readAllLines: OutOfMemoryError (Java heap space)
streamed 4,000,000 rows, total price 19,980,000.00
failed withreadAllLines . RaisingOutOfMemoryError: Java heap space only moves the failure to the next, larger file, and every concurrent request that loads a file needs that memory at the same time.-Xmx- The streaming loop keeps only the current line, so the same 64 MB heap handled all 4 million rows, and the total matches the expected value exactly.
- Each line read by
becomes garbage as soon as the next one is read. Young-generation collection handles such short-lived objects cheaply, so streaming creates many objects without causing long pauses.readLine - The program catches
only to show it. After this error the JVM may be in a bad state; design for streaming instead of catching it.OutOfMemoryError
What a loaded line costs on the heap
Why did 108 MB not fit into 64 MB, and how big would the heap have to be? The program loads a smaller file of 500,000 lines of the same kind and measures the heap before and after, with garbage collection forced so that only live objects are counted.
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
public class MemoryPerLine {
static long usedHeap() {
Runtime rt = Runtime.getRuntime();
for (int i = 0; i < 3; i++) {
System.gc();
}
return rt.totalMemory() - rt.freeMemory();
}
public static void main(String[] args) throws IOException {
Path csv = Files.createTempFile("sample", ".csv");
int rows = 500_000;
try (BufferedWriter w = Files.newBufferedWriter(csv)) {
for (int i = 1; i <= rows; i++) {
w.write(i + ",product-" + i + "," + price(i) + "\n");
}
}
long fileBytes = Files.size(csv);
long before = usedHeap();
List<String> lines = Files.readAllLines(csv);
long after = usedHeap();
long heapBytes = after - before;
System.out.printf("lines: %,d, file: %.1f MB%n", lines.size(), fileBytes / 1e6);
System.out.printf("heap used by the list: %.1f MB%n", heapBytes / 1e6);
System.out.printf("per line: %d bytes of text in the file, about %d bytes on the heap%n",
fileBytes / rows, heapBytes / rows);
System.out.printf("heap / file size: about %.0fx%n", (double) heapBytes / fileBytes);
System.out.println("first line still reachable: " + lines.get(0));
Files.delete(csv);
}
// 0.00, 0.01, ... 9.99, then again from 0.00: always two decimals
static String price(int i) {
int cents = i % 1000;
return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
}
}
Output:
lines: 500,000, file: 13.3 MB
heap used by the list: 38.6 MB
per line: 26 bytes of text in the file, about 77 bytes on the heap
heap / file size: about 3x
first line still reachable: 1,product-1,0.01
Each line is 26 bytes in the file, including the newline, but about 76 bytes on the heap, nearly three times as much. On a 64-bit HotSpot JVM with compressed pointers (the default for heaps under 32 GB), that is:
- 24 bytes for the
object itself: header, the reference to its array, the cached hash code and two small flag fields.String - 16 bytes of array header plus 25 characters, rounded up to a multiple of 8: 48 bytes for the
. Since Java 9 (JEP 254, compact strings), text that fits in Latin-1 uses 1 byte per character; any other character, such as Chinese text, makes the whole string 2 bytes per character.byte[] - 4 bytes for the reference in the
.ArrayList
Short lines make it worse, because the 40 bytes of fixed overhead dominate. So the 108 MB file would need around 300 MB of heap, plus the extra copy the ArrayList
Where the time goes: reading versus parsing
When a large-file job is slow, the first question is whether the reading or the per-line work is slow. The program reads the same 2-million-line file five times with a BufferedReader
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
public class ParseCost {
interface LineWork {
double apply(String line);
}
static double run(Path p, LineWork work) throws IOException {
double sum = 0;
try (BufferedReader r = Files.newBufferedReader(p)) {
String line;
while ((line = r.readLine()) != null) {
sum += work.apply(line);
}
}
return sum;
}
static void measure(String label, Path p, LineWork work) throws IOException {
long best = Long.MAX_VALUE;
double result = 0;
for (int i = 0; i < 5; i++) {
long start = System.nanoTime();
result = run(p, work);
best = Math.min(best, System.nanoTime() - start);
}
System.out.printf("%-34s %5d ms result %,.2f%n", label, best / 1_000_000, result);
}
public static void main(String[] args) throws IOException {
Path csv = Files.createTempFile("orders", ".csv");
try (BufferedWriter w = Files.newBufferedWriter(csv)) {
for (int i = 1; i <= 2_000_000; i++) {
w.write(i + ",product-" + i + "," + price(i) + "\n");
}
}
System.out.printf("file: %d MB, 2,000,000 lines, best of 5%n",
Files.size(csv) / 1_000_000);
run(csv, line -> line.length()); // warm-up, not measured
measure("readLine only", csv, line -> 0);
measure("readLine + String.split", csv,
line -> Double.parseDouble(line.split(",")[2]));
measure("readLine + split with a regex", csv,
line -> Double.parseDouble(line.split("\\s*,\\s*")[2]));
measure("readLine + lastIndexOf/substring", csv,
line -> Double.parseDouble(line.substring(line.lastIndexOf(',') + 1)));
measure("readLine + manual digit parsing", csv, ParseCost::priceInCents);
Files.delete(csv);
}
// "123,product-123,1.23" -> 1.23, read without creating a substring
static double priceInCents(String line) {
long cents = 0;
for (int i = line.lastIndexOf(',') + 1; i < line.length(); i++) {
char c = line.charAt(i);
if (c != '.') {
cents = cents * 10 + (c - '0');
}
}
return cents / 100.0;
}
// 0.00, 0.01, ... 9.99, then again from 0.00: always two decimals
static String price(int i) {
int cents = i % 1000;
return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
}
}
Output (rough comparison on a 2-CPU Linux VM, JDK 21; absolute times will differ on yours):
file: 55 MB, 2,000,000 lines, best of 5
readLine only 71 ms result 0.00
readLine + String.split 277 ms result 9,990,000.00
readLine + split with a regex 858 ms result 9,990,000.00
readLine + lastIndexOf/substring 121 ms result 9,990,000.00
readLine + manual digit parsing 80 ms result 9,990,000.00
- Reading and splitting the file into 2 million
lines is the baseline. Everything above it is the cost of parsing.String made the job two to four times slower than the baseline. It creates an array and three new strings per line, of which the program uses one.split(",") was the slowest by far, around ten times the baseline.split("\\s*,\\s*") has a fast path for a single character that is not a regex metacharacter; any other pattern compiles aString.split on every call and runs the regex engine. If you need a regex, compile it once withPattern outside the loop.Pattern.compile andlastIndexOf create only the one string that is needed, and parsing the digits by hand creates none, which brought the total close to the cost of reading alone.substring
All five variants computed the same total, 9,990,000.00. The lesson is to measure before changing the reading code: replacing BufferedReader
Buffer size for byte streams
For binary data, copying and checksums, you read bytes into an array yourself, and the array size decides how many system calls are made. The program reads a 200 MB file of random bytes with buffer sizes from 64 bytes to 16 MB, after one warm-up read that also puts the file into the operating system's cache.
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;
public class BufferSizes {
static long readWith(Path p, int bufferSize) throws IOException {
byte[] buf = new byte[bufferSize];
long calls = 0;
try (InputStream in = Files.newInputStream(p)) {
while (in.read(buf) != -1) {
calls++;
}
}
return calls;
}
public static void main(String[] args) throws IOException {
Path file = Files.createTempFile("blob", ".bin");
byte[] block = new byte[1 << 20];
new Random(42).nextBytes(block);
try (OutputStream out = Files.newOutputStream(file)) {
for (int i = 0; i < 200; i++) {
out.write(block);
}
}
long size = Files.size(file);
System.out.printf("file: %d MB, best of 5 per buffer size%n", size >> 20);
readWith(file, 1 << 16); // warm-up and fill the OS cache
for (int bufferSize : new int[] {64, 512, 4096, 8192, 65_536, 1 << 20, 16 << 20}) {
long best = Long.MAX_VALUE;
long calls = 0;
for (int r = 0; r < 5; r++) {
long start = System.nanoTime();
calls = readWith(file, bufferSize);
best = Math.min(best, System.nanoTime() - start);
}
System.out.printf("buffer %,10d bytes: %,9d read calls %6d ms %,6.0f MB/s%n",
bufferSize, calls, best / 1_000_000, size / 1e6 / (best / 1e9));
}
Files.delete(file);
}
}
Output (rough comparison on the same machine; best of 5):
file: 200 MB, best of 5 per buffer size
buffer 64 bytes: 3,276,800 read calls 952 ms 220 MB/s
buffer 512 bytes: 409,600 read calls 135 ms 1,550 MB/s
buffer 4,096 bytes: 51,200 read calls 34 ms 6,003 MB/s
buffer 8,192 bytes: 25,600 read calls 28 ms 7,456 MB/s
buffer 65,536 bytes: 3,200 read calls 26 ms 7,974 MB/s
buffer 1,048,576 bytes: 200 read calls 29 ms 7,021 MB/s
buffer 16,777,216 bytes: 13 read calls 43 ms 4,829 MB/s
- With 64-byte reads, the program made 3,276,800 system calls, and the cost of each call dominated. Going to 512 bytes and then 4 KB cut the time several times over.
- By 64 KB the gain had flattened out, and a few runs showed 8 KB already as fast. The JDK's own buffers are in this range:
andBufferedInputStream use 8,192, andBufferedReader uses 16,384.InputStream.transferTo - Buffers of 1 MB were no faster than 64 KB, and 16 MB was clearly slower. A buffer that large no longer fits in the CPU caches, and the JDK copies each read through a temporary native buffer of the same size.
Use 8 KB to 64 KB. A bigger buffer does not make a large file faster to read. The
Files over 2 GB: arrays, mapping windows and positional reads
The program creates a 3 GB file and tries three ways to get at its last bytes. The file is sparse: setLength
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
public class Over2GB {
static final long GB = 1L << 30;
public static void main(String[] args) throws IOException {
// A 3 GB sparse file: setLength reserves the size without writing 3 GB of zeros
Path file = Files.createTempFile("huge", ".bin");
try (RandomAccessFile raf = new RandomAccessFile(file.toFile(), "rw")) {
raf.setLength(3 * GB);
raf.seek(3 * GB - 4);
raf.write("END\n".getBytes(StandardCharsets.US_ASCII));
}
System.out.printf("file size: %,d bytes (Integer.MAX_VALUE is %,d)%n",
Files.size(file), Integer.MAX_VALUE);
try {
Files.readAllBytes(file);
} catch (OutOfMemoryError e) {
System.out.println("readAllBytes: OutOfMemoryError: " + e.getMessage());
}
try (FileChannel ch = FileChannel.open(file, StandardOpenOption.READ)) {
try {
ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size());
} catch (IllegalArgumentException e) {
System.out.println("map whole file: IllegalArgumentException: " + e.getMessage());
}
// Map it in windows of at most 1 GB and visit the last byte of each window
long windows = 0;
for (long pos = 0; pos < ch.size(); pos += GB) {
long len = Math.min(GB, ch.size() - pos);
MappedByteBuffer window = ch.map(FileChannel.MapMode.READ_ONLY, pos, len);
window.get((int) (len - 1));
windows++;
}
System.out.println("mapped in " + windows + " windows of up to 1 GB");
// Or read the bytes you need at a long position, with no mapping at all
ByteBuffer tail = ByteBuffer.allocate(4);
ch.read(tail, ch.size() - 4);
System.out.print("last 4 bytes via read(buf, position): "
+ new String(tail.array(), StandardCharsets.US_ASCII));
}
Files.delete(file);
}
}
Output:
file size: 3,221,225,472 bytes (Integer.MAX_VALUE is 2,147,483,647)
readAllBytes: OutOfMemoryError: Required array size too large
map whole file: IllegalArgumentException: Size exceeds Integer.MAX_VALUE
mapped in 3 windows of up to 1 GB
last 4 bytes via read(buf, position): END
threwFiles.readAllBytes . That message does not mean the heap is too small; it means the file has more bytes than any Java array can hold. The heap was 256 MB, but the error would be the same with 64 GB.OutOfMemoryError: Required array size too large for the whole file threwmap . AIllegalArgumentException: Size exceeds Integer.MAX_VALUE usesMappedByteBuffer positions, so one mapping covers at most 2 GB.int- Mapping the file in three windows of up to 1 GB works. Offsets passed to
aremap , so windows can start anywhere. Records that cross a window boundary need care; overlapping the windows by one record length is a common fix.long takes aFileChannel.read(buffer, position) position and needs no mapping. For reading a few known places in a huge file, it is the simplest option.long works the same way; see theRandomAccessFile.seekRandomAccessFile page .
Since Java 22 (JEP 454, the Foreign Function and Memory API), FileChannel.map(mode, offset, size, arena)MemorySegmentlong
Processing byte ranges in parallel
When parsing is the bottleneck and there are several cores, the file can be split into byte ranges, one per thread. Each boundary is moved forward to just after the next newline, so that no line is cut in two. The program sums the price column of a 4-million-line file four ways and prints the boundaries for a split into four ranges.
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.stream.Stream;
public class ParallelChunks {
// Sum of the last column in cents, parsing bytes in [start, end) of a mapped region
static long sumChunk(Path p, long start, long end) throws IOException {
try (FileChannel ch = FileChannel.open(p, StandardOpenOption.READ)) {
MappedByteBuffer buf = ch.map(FileChannel.MapMode.READ_ONLY, start, end - start);
long total = 0;
long value = 0;
for (int i = 0; i < buf.limit(); i++) {
byte b = buf.get(i);
if (b == ',') {
value = 0; // only the last field survives
} else if (b == '\n') {
total += value;
value = 0;
} else if (b >= '0' && b <= '9') {
value = value * 10 + (b - '0');
}
}
return total;
}
}
// Splits the file into n ranges that each end just after a '\n'
static List<long[]> ranges(Path p, int n) throws IOException {
List<long[]> result = new ArrayList<>();
try (FileChannel ch = FileChannel.open(p, StandardOpenOption.READ)) {
long size = ch.size();
long start = 0;
for (int i = 1; i <= n; i++) {
long end = i == n ? size : Math.max(start, size * i / n);
MappedByteBuffer probe = ch.map(FileChannel.MapMode.READ_ONLY, end,
Math.min(200, size - end));
while (end < size && probe.get() != '\n') {
end++;
}
end = Math.min(size, end + (end < size ? 1 : 0));
result.add(new long[] {start, end});
start = end;
}
}
return result;
}
static long chunked(Path p, int threads) throws Exception {
ExecutorService pool = Executors.newFixedThreadPool(threads);
try {
List<Future<Long>> parts = new ArrayList<>();
for (long[] r : ranges(p, threads)) {
parts.add(pool.submit(() -> sumChunk(p, r[0], r[1])));
}
long total = 0;
for (Future<Long> f : parts) {
total += f.get();
}
return total;
} finally {
pool.shutdown();
}
}
static long readerLoop(Path p) throws IOException {
long total = 0;
try (BufferedReader r = Files.newBufferedReader(p)) {
String line;
while ((line = r.readLine()) != null) {
total += cents(line);
}
}
return total;
}
static long parallelLines(Path p) throws IOException {
try (Stream<String> lines = Files.lines(p)) {
return lines.parallel().mapToLong(ParallelChunks::cents).sum();
}
}
static long cents(String line) {
String price = line.substring(line.lastIndexOf(',') + 1);
return Long.parseLong(price.replace(".", ""));
}
interface Job {
long run() throws Exception;
}
static void measure(String label, Job job) throws Exception {
long best = Long.MAX_VALUE;
long result = 0;
for (int i = 0; i < 3; i++) {
long start = System.nanoTime();
result = job.run();
best = Math.min(best, System.nanoTime() - start);
}
System.out.printf("%-32s %5d ms total %,d cents%n", label, best / 1_000_000, result);
}
public static void main(String[] args) throws Exception {
Path csv = Files.createTempFile("orders", ".csv");
try (BufferedWriter w = Files.newBufferedWriter(csv)) {
for (int i = 1; i <= 4_000_000; i++) {
w.write(i + ",product-" + i + "," + price(i) + "\n");
}
}
System.out.printf("file: %d MB, %d CPUs, best of 3%n", Files.size(csv) / 1_000_000,
Runtime.getRuntime().availableProcessors());
readerLoop(csv); // warm-up
chunked(csv, 2);
for (long[] r : ranges(csv, 4)) {
System.out.printf(" range %,12d to %,12d%n", r[0], r[1]);
}
measure("BufferedReader loop", () -> readerLoop(csv));
measure("Files.lines().parallel()", () -> parallelLines(csv));
measure("byte ranges, 1 thread", () -> chunked(csv, 1));
measure("byte ranges, 2 threads", () -> chunked(csv, 2));
Files.delete(csv);
}
static String price(int i) {
int cents = i % 1000;
return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
}
}
Output (rough comparison, same 2-CPU machine):
file: 113 MB, 2 CPUs, best of 3
range 0 to 28,444,451
range 28,444,451 to 56,888,898
range 56,888,898 to 85,333,345
range 85,333,345 to 113,777,792
BufferedReader loop 283 ms total 1,998,000,000 cents
Files.lines().parallel() 271 ms total 1,998,000,000 cents
byte ranges, 1 thread 109 ms total 1,998,000,000 cents
byte ranges, 2 threads 111 ms total 1,998,000,000 cents
- The four ranges have almost equal sizes, and each boundary sits just after a newline, so every line belongs to exactly one range. All four methods gave the same total of 1,998,000,000 cents.
- Most of the gain came from parsing bytes directly in a mapped buffer, with no
per line: with one thread it was two to three times faster than theString loop.BufferedReader - Whether two threads helped depended on whether the second CPU was free. On this shared 2-CPU machine, repeated runs ranged from no gain at all to nearly half the time. On a machine with more cores the split scales further, until memory bandwidth or the disk becomes the limit.
needs no extra code. Since Java 9 it splits the file by byte ranges itself when the charset is UTF-8, ISO-8859-1 or US-ASCII. It was no faster than the byte parsing in any run, and sometimes barely faster than the sequential loop, because it still creates aFiles.lines().parallel() per line.String
Parsing raw bytes this way only works for simple, ASCII-based formats. For anything with quoting, escapes or multi-byte characters, decode the text first and use a proper parser.
Choosing a strategy
| Situation | Use |
|---|---|
| Line-based text of any size | Files.newBufferedReaderFiles.lines |
| File fits easily in the heap (well under a fifth of it) | Files.readAllLinesreadString |
| Job is slow | Measure the parsing first; avoid regex split |
| Binary data, copying, checksums | InputStream.readtransferTo |
| A few known positions in a huge file | FileChannel.read(buf, position)RandomAccessFile.seek |
| Scanning bytes of a file over 2 GB | FileChannel.mapMemorySegment |
| CPU-heavy parsing on many cores | Newline-aligned byte ranges, one per thread, or Files.lines().parallel() |
| Compressed input | Stream through GZIPInputStream |
| Last lines of a growing log | Seek near the end; see |