How to handle large files in Java

A file becomes "large" in Java at three points: when it no longer fits in the heap, when it passes the 2 GB limit of arrays and buffers, and when reading it takes long enough to matter. This page shows each limit on real files, measures what a loaded line costs in memory, where the time of a big job really goes, and how to split a file across threads.

Contents

What makes a file large in Java

Size alone does not decide whether a file is a problem. What matters is how it compares with three limits:

  • The heap: loading a file creates Java objects, and they take several times more memory than the bytes on disk. A 100 MB CSV file can need 300 MB of heap once it is a list of strings.
  • The 2 GB index limit: arrays, String, ByteBuffer and MappedByteBuffer are indexed with an int, so none of them can hold more than 2,147,483,647 elements. A 3 GB file cannot be read into one byte[] or mapped as one buffer, however much memory the machine has.
  • Time: reading itself is fast, often several GB per second from the operating system's cache. In a slow job the time usually goes into what the program does with each line, not into the reading.

The strategies for large files follow from these limits. Most avoid holding the file at all; the rest work around the 2 GB limit or spread the work over several cores.

Strategies for large files
StrategySinceMemoryBest for
Stream lines: BufferedReader, Files.linesJava 1.1, Java 8One lineLine-based text of any size
Stream bytes with a large buffer: InputStream.read(byte[])Java 1.0One bufferBinary files, copying, checksums
Positional reads: FileChannel.read(buf, position)Java 1.4One bufferAny part of a file of any size
Memory mapping in windows: FileChannel.mapJava 1.4Outside the heap, paged in by the OSByte-level scanning and random access
One mapping of any size: FileChannel.map(..., Arena)Java 22 (JEP 454)Outside the heapFiles over 2 GB without windows
Byte ranges processed in parallelAnyOne buffer per threadCPU-heavy parsing on many cores

The basic reading APIs and their relative speed on an ordinary file are covered on reading a text file. This page concentrates on what changes when the file is big. All programs were compiled and run on JDK 21; their test files are generated on the fly, so the numbers are reproducible.

Loading versus streaming a file bigger than the heap

The first program creates a CSV file with 4 million rows, about 108 MB, and runs with the heap limited to 64 MB (-Xmx64m), so the file is bigger than the whole heap. It first tries to load every line with Files.readAllLines, then sums the price column with a BufferedReader. The prices cycle through 0.00 to 9.99, so the expected total is easy to check: 4,000 cycles of 4,995.00 make 19,980,000.

Java
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;

public class LargeFileMemory {

    // Writes a header and rows like "123,product-123,1.23"
    static Path createCsv(int rows) throws IOException {
        Path file = Files.createTempFile("large", ".csv");
        try (BufferedWriter w = Files.newBufferedWriter(file)) {
            w.write("id,name,price\n");
            for (int i = 1; i <= rows; i++) {
                w.write(i + ",product-" + i + "," + price(i) + "\n");
            }
        }
        return file;
    }

    public static void main(String[] args) throws IOException {
        long maxHeapMb = Runtime.getRuntime().maxMemory() / (1024 * 1024);
        Path csv = createCsv(4_000_000);
        System.out.printf("max heap: %d MB, file size: %d MB%n", maxHeapMb,
                Files.size(csv) / (1024 * 1024));

        // Loading every line at once needs far more memory than the file size
        try {
            List<String> all = Files.readAllLines(csv);
            System.out.println("readAllLines: loaded " + all.size() + " lines");
        } catch (OutOfMemoryError e) {
            System.out.println("readAllLines: OutOfMemoryError (" + e.getMessage() + ")");
        }

        // Streaming keeps only the current line in memory
        long rows = 0;
        double total = 0;
        try (BufferedReader r = Files.newBufferedReader(csv)) {
            r.readLine();                                   // skip the header
            String line;
            while ((line = r.readLine()) != null) {
                total += Double.parseDouble(line.substring(line.lastIndexOf(',') + 1));
                rows++;
            }
        }
        System.out.printf("streamed %,d rows, total price %,.2f%n", rows, total);
        Files.delete(csv);
    }

    // 0.00, 0.01, ... 9.99, then again from 0.00: always two decimals
    static String price(int i) {
        int cents = i % 1000;
        return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
    }
}

Output:

Output
max heap: 64 MB, file size: 108 MB
readAllLines: OutOfMemoryError (Java heap space)
streamed 4,000,000 rows, total price 19,980,000.00
  • readAllLines failed with OutOfMemoryError: Java heap space. Raising -Xmx only moves the failure to the next, larger file, and every concurrent request that loads a file needs that memory at the same time.
  • The streaming loop keeps only the current line, so the same 64 MB heap handled all 4 million rows, and the total matches the expected value exactly.
  • Each line read by readLine becomes garbage as soon as the next one is read. Young-generation collection handles such short-lived objects cheaply, so streaming creates many objects without causing long pauses.
  • The program catches OutOfMemoryError only to show it. After this error the JVM may be in a bad state; design for streaming instead of catching it.

What a loaded line costs on the heap

Why did 108 MB not fit into 64 MB, and how big would the heap have to be? The program loads a smaller file of 500,000 lines of the same kind and measures the heap before and after, with garbage collection forced so that only live objects are counted.

Java
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;

public class MemoryPerLine {

    static long usedHeap() {
        Runtime rt = Runtime.getRuntime();
        for (int i = 0; i < 3; i++) {
            System.gc();
        }
        return rt.totalMemory() - rt.freeMemory();
    }

    public static void main(String[] args) throws IOException {
        Path csv = Files.createTempFile("sample", ".csv");
        int rows = 500_000;
        try (BufferedWriter w = Files.newBufferedWriter(csv)) {
            for (int i = 1; i <= rows; i++) {
                w.write(i + ",product-" + i + "," + price(i) + "\n");
            }
        }
        long fileBytes = Files.size(csv);

        long before = usedHeap();
        List<String> lines = Files.readAllLines(csv);
        long after = usedHeap();
        long heapBytes = after - before;

        System.out.printf("lines: %,d, file: %.1f MB%n", lines.size(), fileBytes / 1e6);
        System.out.printf("heap used by the list: %.1f MB%n", heapBytes / 1e6);
        System.out.printf("per line: %d bytes of text in the file, about %d bytes on the heap%n",
                fileBytes / rows, heapBytes / rows);
        System.out.printf("heap / file size: about %.0fx%n", (double) heapBytes / fileBytes);
        System.out.println("first line still reachable: " + lines.get(0));
        Files.delete(csv);
    }

    // 0.00, 0.01, ... 9.99, then again from 0.00: always two decimals
    static String price(int i) {
        int cents = i % 1000;
        return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
    }
}

Output:

Output
lines: 500,000, file: 13.3 MB
heap used by the list: 38.6 MB
per line: 26 bytes of text in the file, about 77 bytes on the heap
heap / file size: about 3x
first line still reachable: 1,product-1,0.01

Each line is 26 bytes in the file, including the newline, but about 76 bytes on the heap, nearly three times as much. On a 64-bit HotSpot JVM with compressed pointers (the default for heaps under 32 GB), that is:

  • 24 bytes for the String object itself: header, the reference to its array, the cached hash code and two small flag fields.
  • 16 bytes of array header plus 25 characters, rounded up to a multiple of 8: 48 bytes for the byte[]. Since Java 9 (JEP 254, compact strings), text that fits in Latin-1 uses 1 byte per character; any other character, such as Chinese text, makes the whole string 2 bytes per character.
  • 4 bytes for the reference in the ArrayList.

Short lines make it worse, because the 40 bytes of fixed overhead dominate. So the 108 MB file would need around 300 MB of heap, plus the extra copy the ArrayList makes each time it grows. A rule of thumb for line-based text: loading takes three to six times the file size.

Where the time goes: reading versus parsing

When a large-file job is slow, the first question is whether the reading or the per-line work is slow. The program reads the same 2-million-line file five times with a BufferedReader and changes only how each line is parsed. Each variant runs 5 times after a warm-up run and the fastest time is shown.

Java
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;

public class ParseCost {

    interface LineWork {
        double apply(String line);
    }

    static double run(Path p, LineWork work) throws IOException {
        double sum = 0;
        try (BufferedReader r = Files.newBufferedReader(p)) {
            String line;
            while ((line = r.readLine()) != null) {
                sum += work.apply(line);
            }
        }
        return sum;
    }

    static void measure(String label, Path p, LineWork work) throws IOException {
        long best = Long.MAX_VALUE;
        double result = 0;
        for (int i = 0; i < 5; i++) {
            long start = System.nanoTime();
            result = run(p, work);
            best = Math.min(best, System.nanoTime() - start);
        }
        System.out.printf("%-34s %5d ms   result %,.2f%n", label, best / 1_000_000, result);
    }

    public static void main(String[] args) throws IOException {
        Path csv = Files.createTempFile("orders", ".csv");
        try (BufferedWriter w = Files.newBufferedWriter(csv)) {
            for (int i = 1; i <= 2_000_000; i++) {
                w.write(i + ",product-" + i + "," + price(i) + "\n");
            }
        }
        System.out.printf("file: %d MB, 2,000,000 lines, best of 5%n",
                Files.size(csv) / 1_000_000);

        run(csv, line -> line.length());                // warm-up, not measured
        measure("readLine only", csv, line -> 0);
        measure("readLine + String.split", csv,
                line -> Double.parseDouble(line.split(",")[2]));
        measure("readLine + split with a regex", csv,
                line -> Double.parseDouble(line.split("\\s*,\\s*")[2]));
        measure("readLine + lastIndexOf/substring", csv,
                line -> Double.parseDouble(line.substring(line.lastIndexOf(',') + 1)));
        measure("readLine + manual digit parsing", csv, ParseCost::priceInCents);
        Files.delete(csv);
    }

    // "123,product-123,1.23" -> 1.23, read without creating a substring
    static double priceInCents(String line) {
        long cents = 0;
        for (int i = line.lastIndexOf(',') + 1; i < line.length(); i++) {
            char c = line.charAt(i);
            if (c != '.') {
                cents = cents * 10 + (c - '0');
            }
        }
        return cents / 100.0;
    }

    // 0.00, 0.01, ... 9.99, then again from 0.00: always two decimals
    static String price(int i) {
        int cents = i % 1000;
        return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
    }
}

Output (rough comparison on a 2-CPU Linux VM, JDK 21; absolute times will differ on yours):

Output
file: 55 MB, 2,000,000 lines, best of 5
readLine only                         71 ms   result 0.00
readLine + String.split              277 ms   result 9,990,000.00
readLine + split with a regex        858 ms   result 9,990,000.00
readLine + lastIndexOf/substring     121 ms   result 9,990,000.00
readLine + manual digit parsing       80 ms   result 9,990,000.00
  • Reading and splitting the file into 2 million String lines is the baseline. Everything above it is the cost of parsing.
  • split(",") made the job two to four times slower than the baseline. It creates an array and three new strings per line, of which the program uses one.
  • split("\\s*,\\s*") was the slowest by far, around ten times the baseline. String.split has a fast path for a single character that is not a regex metacharacter; any other pattern compiles a Pattern on every call and runs the regex engine. If you need a regex, compile it once with Pattern.compile outside the loop.
  • lastIndexOf and substring create only the one string that is needed, and parsing the digits by hand creates none, which brought the total close to the cost of reading alone.

All five variants computed the same total, 9,990,000.00. The lesson is to measure before changing the reading code: replacing BufferedReader with something faster could save at most the baseline time, while fixing the parsing saved far more.

Buffer size for byte streams

For binary data, copying and checksums, you read bytes into an array yourself, and the array size decides how many system calls are made. The program reads a 200 MB file of random bytes with buffer sizes from 64 bytes to 16 MB, after one warm-up read that also puts the file into the operating system's cache.

Java
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;

public class BufferSizes {

    static long readWith(Path p, int bufferSize) throws IOException {
        byte[] buf = new byte[bufferSize];
        long calls = 0;
        try (InputStream in = Files.newInputStream(p)) {
            while (in.read(buf) != -1) {
                calls++;
            }
        }
        return calls;
    }

    public static void main(String[] args) throws IOException {
        Path file = Files.createTempFile("blob", ".bin");
        byte[] block = new byte[1 << 20];
        new Random(42).nextBytes(block);
        try (OutputStream out = Files.newOutputStream(file)) {
            for (int i = 0; i < 200; i++) {
                out.write(block);
            }
        }
        long size = Files.size(file);
        System.out.printf("file: %d MB, best of 5 per buffer size%n", size >> 20);

        readWith(file, 1 << 16);                        // warm-up and fill the OS cache
        for (int bufferSize : new int[] {64, 512, 4096, 8192, 65_536, 1 << 20, 16 << 20}) {
            long best = Long.MAX_VALUE;
            long calls = 0;
            for (int r = 0; r < 5; r++) {
                long start = System.nanoTime();
                calls = readWith(file, bufferSize);
                best = Math.min(best, System.nanoTime() - start);
            }
            System.out.printf("buffer %,10d bytes: %,9d read calls %6d ms  %,6.0f MB/s%n",
                    bufferSize, calls, best / 1_000_000, size / 1e6 / (best / 1e9));
        }
        Files.delete(file);
    }
}

Output (rough comparison on the same machine; best of 5):

Output
file: 200 MB, best of 5 per buffer size
buffer         64 bytes: 3,276,800 read calls    952 ms     220 MB/s
buffer        512 bytes:   409,600 read calls    135 ms   1,550 MB/s
buffer      4,096 bytes:    51,200 read calls     34 ms   6,003 MB/s
buffer      8,192 bytes:    25,600 read calls     28 ms   7,456 MB/s
buffer     65,536 bytes:     3,200 read calls     26 ms   7,974 MB/s
buffer  1,048,576 bytes:       200 read calls     29 ms   7,021 MB/s
buffer 16,777,216 bytes:        13 read calls     43 ms   4,829 MB/s
  • With 64-byte reads, the program made 3,276,800 system calls, and the cost of each call dominated. Going to 512 bytes and then 4 KB cut the time several times over.
  • By 64 KB the gain had flattened out, and a few runs showed 8 KB already as fast. The JDK's own buffers are in this range: BufferedInputStream and BufferedReader use 8,192, and InputStream.transferTo uses 16,384.
  • Buffers of 1 MB were no faster than 64 KB, and 16 MB was clearly slower. A buffer that large no longer fits in the CPU caches, and the JDK copies each read through a temporary native buffer of the same size.

Use 8 KB to 64 KB. A bigger buffer does not make a large file faster to read. The buffered streams page explains what the buffering classes add on top.

Files over 2 GB: arrays, mapping windows and positional reads

The program creates a 3 GB file and tries three ways to get at its last bytes. The file is sparse: setLength sets its size without writing 3 GB of zeros, so on file systems that support sparse files, such as ext4 on Linux, it takes almost no disk space and is created instantly.

Java
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;

public class Over2GB {

    static final long GB = 1L << 30;

    public static void main(String[] args) throws IOException {
        // A 3 GB sparse file: setLength reserves the size without writing 3 GB of zeros
        Path file = Files.createTempFile("huge", ".bin");
        try (RandomAccessFile raf = new RandomAccessFile(file.toFile(), "rw")) {
            raf.setLength(3 * GB);
            raf.seek(3 * GB - 4);
            raf.write("END\n".getBytes(StandardCharsets.US_ASCII));
        }
        System.out.printf("file size: %,d bytes (Integer.MAX_VALUE is %,d)%n",
                Files.size(file), Integer.MAX_VALUE);

        try {
            Files.readAllBytes(file);
        } catch (OutOfMemoryError e) {
            System.out.println("readAllBytes:    OutOfMemoryError: " + e.getMessage());
        }

        try (FileChannel ch = FileChannel.open(file, StandardOpenOption.READ)) {
            try {
                ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size());
            } catch (IllegalArgumentException e) {
                System.out.println("map whole file: IllegalArgumentException: " + e.getMessage());
            }

            // Map it in windows of at most 1 GB and visit the last byte of each window
            long windows = 0;
            for (long pos = 0; pos < ch.size(); pos += GB) {
                long len = Math.min(GB, ch.size() - pos);
                MappedByteBuffer window = ch.map(FileChannel.MapMode.READ_ONLY, pos, len);
                window.get((int) (len - 1));
                windows++;
            }
            System.out.println("mapped in " + windows + " windows of up to 1 GB");

            // Or read the bytes you need at a long position, with no mapping at all
            ByteBuffer tail = ByteBuffer.allocate(4);
            ch.read(tail, ch.size() - 4);
            System.out.print("last 4 bytes via read(buf, position): "
                    + new String(tail.array(), StandardCharsets.US_ASCII));
        }
        Files.delete(file);
    }
}

Output:

Output
file size: 3,221,225,472 bytes (Integer.MAX_VALUE is 2,147,483,647)
readAllBytes:    OutOfMemoryError: Required array size too large
map whole file: IllegalArgumentException: Size exceeds Integer.MAX_VALUE
mapped in 3 windows of up to 1 GB
last 4 bytes via read(buf, position): END
  • Files.readAllBytes threw OutOfMemoryError: Required array size too large. That message does not mean the heap is too small; it means the file has more bytes than any Java array can hold. The heap was 256 MB, but the error would be the same with 64 GB.
  • map for the whole file threw IllegalArgumentException: Size exceeds Integer.MAX_VALUE. A MappedByteBuffer uses int positions, so one mapping covers at most 2 GB.
  • Mapping the file in three windows of up to 1 GB works. Offsets passed to map are long, so windows can start anywhere. Records that cross a window boundary need care; overlapping the windows by one record length is a common fix.
  • FileChannel.read(buffer, position) takes a long position and needs no mapping. For reading a few known places in a huge file, it is the simplest option. RandomAccessFile.seek works the same way; see the RandomAccessFile page.

Since Java 22 (JEP 454, the Foreign Function and Memory API), FileChannel.map(mode, offset, size, arena) returns a MemorySegment with long offsets, so a 3 GB file can be mapped in one piece. In Java 21 that method is still a preview API. See foreign memory and NIO channels for how mapping works.

Processing byte ranges in parallel

When parsing is the bottleneck and there are several cores, the file can be split into byte ranges, one per thread. Each boundary is moved forward to just after the next newline, so that no line is cut in two. The program sums the price column of a 4-million-line file four ways and prints the boundaries for a split into four ranges.

Java
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.stream.Stream;

public class ParallelChunks {

    // Sum of the last column in cents, parsing bytes in [start, end) of a mapped region
    static long sumChunk(Path p, long start, long end) throws IOException {
        try (FileChannel ch = FileChannel.open(p, StandardOpenOption.READ)) {
            MappedByteBuffer buf = ch.map(FileChannel.MapMode.READ_ONLY, start, end - start);
            long total = 0;
            long value = 0;
            for (int i = 0; i < buf.limit(); i++) {
                byte b = buf.get(i);
                if (b == ',') {
                    value = 0;                          // only the last field survives
                } else if (b == '\n') {
                    total += value;
                    value = 0;
                } else if (b >= '0' && b <= '9') {
                    value = value * 10 + (b - '0');
                }
            }
            return total;
        }
    }

    // Splits the file into n ranges that each end just after a '\n'
    static List<long[]> ranges(Path p, int n) throws IOException {
        List<long[]> result = new ArrayList<>();
        try (FileChannel ch = FileChannel.open(p, StandardOpenOption.READ)) {
            long size = ch.size();
            long start = 0;
            for (int i = 1; i <= n; i++) {
                long end = i == n ? size : Math.max(start, size * i / n);
                MappedByteBuffer probe = ch.map(FileChannel.MapMode.READ_ONLY, end,
                        Math.min(200, size - end));
                while (end < size && probe.get() != '\n') {
                    end++;
                }
                end = Math.min(size, end + (end < size ? 1 : 0));
                result.add(new long[] {start, end});
                start = end;
            }
        }
        return result;
    }

    static long chunked(Path p, int threads) throws Exception {
        ExecutorService pool = Executors.newFixedThreadPool(threads);
        try {
            List<Future<Long>> parts = new ArrayList<>();
            for (long[] r : ranges(p, threads)) {
                parts.add(pool.submit(() -> sumChunk(p, r[0], r[1])));
            }
            long total = 0;
            for (Future<Long> f : parts) {
                total += f.get();
            }
            return total;
        } finally {
            pool.shutdown();
        }
    }

    static long readerLoop(Path p) throws IOException {
        long total = 0;
        try (BufferedReader r = Files.newBufferedReader(p)) {
            String line;
            while ((line = r.readLine()) != null) {
                total += cents(line);
            }
        }
        return total;
    }

    static long parallelLines(Path p) throws IOException {
        try (Stream<String> lines = Files.lines(p)) {
            return lines.parallel().mapToLong(ParallelChunks::cents).sum();
        }
    }

    static long cents(String line) {
        String price = line.substring(line.lastIndexOf(',') + 1);
        return Long.parseLong(price.replace(".", ""));
    }

    interface Job {
        long run() throws Exception;
    }

    static void measure(String label, Job job) throws Exception {
        long best = Long.MAX_VALUE;
        long result = 0;
        for (int i = 0; i < 3; i++) {
            long start = System.nanoTime();
            result = job.run();
            best = Math.min(best, System.nanoTime() - start);
        }
        System.out.printf("%-32s %5d ms   total %,d cents%n", label, best / 1_000_000, result);
    }

    public static void main(String[] args) throws Exception {
        Path csv = Files.createTempFile("orders", ".csv");
        try (BufferedWriter w = Files.newBufferedWriter(csv)) {
            for (int i = 1; i <= 4_000_000; i++) {
                w.write(i + ",product-" + i + "," + price(i) + "\n");
            }
        }
        System.out.printf("file: %d MB, %d CPUs, best of 3%n", Files.size(csv) / 1_000_000,
                Runtime.getRuntime().availableProcessors());
        readerLoop(csv);                                // warm-up
        chunked(csv, 2);
        for (long[] r : ranges(csv, 4)) {
            System.out.printf("  range %,12d to %,12d%n", r[0], r[1]);
        }
        measure("BufferedReader loop", () -> readerLoop(csv));
        measure("Files.lines().parallel()", () -> parallelLines(csv));
        measure("byte ranges, 1 thread", () -> chunked(csv, 1));
        measure("byte ranges, 2 threads", () -> chunked(csv, 2));
        Files.delete(csv);
    }

    static String price(int i) {
        int cents = i % 1000;
        return cents / 100 + "." + (cents % 100 < 10 ? "0" : "") + cents % 100;
    }
}

Output (rough comparison, same 2-CPU machine):

Output
file: 113 MB, 2 CPUs, best of 3
  range            0 to   28,444,451
  range   28,444,451 to   56,888,898
  range   56,888,898 to   85,333,345
  range   85,333,345 to  113,777,792
BufferedReader loop                283 ms   total 1,998,000,000 cents
Files.lines().parallel()           271 ms   total 1,998,000,000 cents
byte ranges, 1 thread              109 ms   total 1,998,000,000 cents
byte ranges, 2 threads             111 ms   total 1,998,000,000 cents
  • The four ranges have almost equal sizes, and each boundary sits just after a newline, so every line belongs to exactly one range. All four methods gave the same total of 1,998,000,000 cents.
  • Most of the gain came from parsing bytes directly in a mapped buffer, with no String per line: with one thread it was two to three times faster than the BufferedReader loop.
  • Whether two threads helped depended on whether the second CPU was free. On this shared 2-CPU machine, repeated runs ranged from no gain at all to nearly half the time. On a machine with more cores the split scales further, until memory bandwidth or the disk becomes the limit.
  • Files.lines().parallel() needs no extra code. Since Java 9 it splits the file by byte ranges itself when the charset is UTF-8, ISO-8859-1 or US-ASCII. It was no faster than the byte parsing in any run, and sometimes barely faster than the sequential loop, because it still creates a String per line.

Parsing raw bytes this way only works for simple, ASCII-based formats. For anything with quoting, escapes or multi-byte characters, decode the text first and use a proper parser.

Choosing a strategy

Approaches for large files
SituationUse
Line-based text of any sizeFiles.newBufferedReader or Files.lines, processing each line as it arrives
File fits easily in the heap (well under a fifth of it)Files.readAllLines or readString is fine
Job is slowMeasure the parsing first; avoid regex split in the loop
Binary data, copying, checksumsInputStream.read with an 8 KB to 64 KB buffer, or transferTo
A few known positions in a huge fileFileChannel.read(buf, position) or RandomAccessFile.seek
Scanning bytes of a file over 2 GBFileChannel.map in windows; on Java 22 and later, one MemorySegment
CPU-heavy parsing on many coresNewline-aligned byte ranges, one per thread, or Files.lines().parallel()
Compressed inputStream through GZIPInputStream; see ZIP and GZIP
Last lines of a growing logSeek near the end; see reading a text file