Package org.microproxy.warc

Class WarcWriter

java.lang.Object

org.microproxy.warc.WarcWriter

All Implemented Interfaces:

Closeable, AutoCloseable

public final class WarcWriter
extends Object
implements Closeable

Appends WARC 1.1 records to files in a directory, starting a new file (each opened by a warcinfo record) when one reaches its size limit. A file is named prefix-timestamp-serial.warc.gz and carries an extra .open suffix while it is being written. Each record is its own gzip member, so readers can seek to any record. Thread-safe; records passed to one write(java.util.List<org.microproxy.warc.WarcWriter.Record>) call stay together.

Constructor Summary

Constructors

Constructor	Description
WarcWriter(Path dir, String prefix, long maxFileSize, boolean gzip, String software)	Creates a WARC writer with file rotation and optional compression.

Method Summary

Modifier and Type	Method	Description
void	close()	Finishes the current file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

WarcWriter

public WarcWriter(Path dir,
 String prefix,
 long maxFileSize,
 boolean gzip,
 String software)
           throws IOException

Creates a WARC writer with file rotation and optional compression.

Parameters:

dir - the directory in which to create WARC files

prefix - the WARC file-name prefix

maxFileSize - start a new file once the current one reaches this many bytes

gzip - compress each record (.warc.gz); otherwise plain .warc

software - the software identification written to warcinfo records

Throws:

IOException - if the output directory cannot be created

Method Details

close

public void close()
           throws IOException

Finishes the current file. Later writes fail.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException