Package org.microproxy.warc

Class WarcReader

java.lang.Object

org.microproxy.warc.WarcReader

All Implemented Interfaces:

Closeable, AutoCloseable

public final class WarcReader
extends Object
implements Closeable

Reads WARC records (versions 1.0 and 1.1) one after another, as WarcRecorder writes them and as other archiving tools do: plain .warc files, or .warc.gz files with one gzip member per record (or one for the whole file). ServerReplay uses it to answer requests from recorded responses.

 try (WarcReader reader = WarcReader.open(Path.of("warcs/microproxy-....warc.gz"))) {
     for (WarcReader.Record r; (r = reader.next()) != null; ) {
         if (r.type().equals("response")) System.out.println(r.field("WARC-Target-URI"));
     }
 }
 

Each record's block is read into memory whole.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	WarcReader.Record	One WARC record: its version line, named fields and block.

Constructor Summary

Constructors

Constructor	Description
WarcReader(InputStream in)	Reads records from in, which is decompressed if it starts with the gzip magic bytes.

Method Summary

Modifier and Type	Method	Description
void	close()
WarcReader.Record	next()	Reads the next record.
static WarcReader	open(Path file)	Opens a WARC file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

WarcReader

public WarcReader(InputStream in)
           throws IOException

Reads records from in, which is decompressed if it starts with the gzip magic bytes.

Parameters:

in - the WARC data

Throws:

IOException - if the stream cannot be read

Method Details

open

public static WarcReader open(Path file)
                       throws IOException

Opens a WARC file.

Parameters:

file - a .warc or .warc.gz file

Returns:

a reader positioned at the first record

Throws:

IOException - if the file cannot be opened

next

public WarcReader.Record next()
                       throws IOException

Reads the next record.

Returns:

the record, or null at the end of the input

Throws:

IOException - if the input is not WARC or ends inside a record

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException