The following are changes currently implemented in development builds of Cyotek WebCopy. Please note that these changes are not final and may be removed prior to the next stable release. If you have any comments about these forthcoming changes, please contact us.

Nightly builds featuring these changes are available from the Downloads page.

Due to changes in how WebCopy determines whether or not to process a given URL there could be differences with how WebCopy 1.9 works against previous versions. Please report any inconsistencies to us!

Added

  • It is now possible to read additional URLs to scan from a text file [#281]
  • Added no-directories, max-redirect and header arguments to CLI
  • Added proxy, proxy-user and proxy-password arguments to CLI [#337]
  • Added input-file argument to CLI [#282]

Changed

  • Adding multiple URLs to scan is now easier using a free text field [#282]
  • Command line tools now report unknown parameters
  • Major reworking of internal decision making logic [#242]
  • New WebCopy projects will default to enabling Brotli compression support
  • New WebCopy projects will default to saving headers

Fixed

  • Only the last argument error was displayed when running command line tools
  • WebCopy will now retry URLs that fail with "The server committed a protocol violation" exceptions
  • If using the default user agent, WebCopy will now try a default browser agent if a 401 response is returned when validating the URL [#382]
  • When issuing a 401 challenge, WebCopy could include additional header information in the realm field

The following changes are a sneak-peak of features that are currently being experimented with in separate branches from the core product. Due to this they are not available in nightly builds (and there is no guarantee they will be merged into the core branch).

Added

  • Crawl state persistence (pause, resume) [#165]

Changed

  • Crawling now uses breadth first when prioritising enqueued items

Download

Download current and archived versions of Cyotek WebCopy

Download

Minimum Requirements

Donate

Donate