Package org.microproxy.warc

Class WarcRecorder

java.lang.Object
org.microproxy.warc.WarcRecorder
All Implemented Interfaces:
Closeable, AutoCloseable, HttpFiltersSource
public final class WarcRecorder extends Object implements HttpFiltersSource, Closeable
Records the traffic between the proxy and servers as WARC 1.1 files (ISO 28500), the format web archives use, so captures can be replayed with tools such as pywb or inspected with warcio. 

 WarcRecorder recorder = WarcRecorder.builder(Path.of("warcs")).build();
 MicroProxy.bootstrap().withFiltersSource(recorder).start();
 ...
 recorder.close();

Each exchange with a server becomes a request record and a response record linked by WARC-Concurrent-To, with block and payload digests, the target URI ( https:// inside intercepted sessions) and the server's IP address when known. Bodies are captured as they stream past, without buffering the exchange: in memory, or in a temporary file once large, up to WarcRecorder.Builder.maxBodySize (beyond that the record is marked WARC-Truncated: length). Messages are recorded as exchanged except that chunked transfer coding is removed and Content-Length states the recorded body length; content codings such as gzip are kept.

Only traffic that reaches a server is recorded: responses from a cache or a filter are not, and neither are tunnelled (CONNECT, not intercepted) bytes. Install the recorder first among the filters to record messages before other filters change them.

  • Method Details

    • builder

      public static WarcRecorder.Builder builder(Path dir)
      Starts a builder with the default settings.
      Parameters:
      dir - the directory in which to create WARC files
      Returns:
      a new builder with default settings

    • recordedExchanges

      public long recordedExchanges()
      Returns exchanges recorded so far.
      Returns:
      exchanges recorded so far

    • failedExchanges

      public long failedExchanges()
      Returns exchanges that could not be written.
      Returns:
      exchanges that could not be written

    • filterRequest

      public HttpFilters filterRequest(HttpRequest originalRequest, FlowContext flowContext)
      Description copied from interface: HttpFiltersSource
      Returns the filters for a request, or null to proxy it unfiltered.
      Specified by:
      filterRequest in interface HttpFiltersSource
      Parameters:
      originalRequest - a copy of the request as received from the client
      flowContext - the client connection the request arrived on
      Returns:
      the per-request filters, or null to leave the request unfiltered

    • close

      public void close() throws IOException
      Finishes the current WARC file.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException