Cyotek WebCopy Upcoming Changes Copy websites locally for offline browsing

The following are changes currently implemented in development builds of Cyotek WebCopy. Please note that these changes are not final and may be removed prior to the next stable release. If you have any comments about these forthcoming changes, please contact us.

Nightly builds featuring these changes are available from the Downloads page.

Added

• Added the ability to read cookies from an external file (#461) (User Manual)
• Added the ability to read cookies from an external file (#462) (User Manual)
• Test URL dialogue now allows configuring cookies
• Added cookie, cookie-jar and discard-session-cookies command line parameters (User Manual)

Changed

• Documentation improvements (#400, #443, #461, #462 untracked)
• Test URL dialogue now uses load on demand for settings pages

Fixed

• Posting a form did not set an appropriate content type (#437)
• Custom headers were not applied when posting forms (#436)
• If a URL was previously skipped but then included in future scans, the original skip reason could be retained
• A blank error message was displayed for Brotli decompression errors (#447)
• One-time project validation checks were ignoring the content encoding settings of the project (which by default is Gzip and Deflate) and were requesting content with Brotli compression (#446)
• Brotli decompression could fail with streams larger than 65535 bytes (#448)
• The URI transformation service incorrectly attempted to add prefixes to email addresses, this in turn caused a crash if the mailto: reference was malformed (#450)
• A crash could occur if a content type header was malformed and was either utf or utf- (#450)
• Fixed an issue where command line arguments sometimes didn't correctly process ambiguous relative arguments that could be a file name or a unqualified URI (#441). As a result of this fix, all URIs provided to command lines must be fully qualified, e.g. https://example.com over example.com
• Fixed a crash that could occur when switching between empty virtual list views during a crawl and items were then subsequently added (#460)
• A crash which occurred loading localised text is no longer fatal (#465). Note that we haven't been able to directly reproduce this, so if you received a crash after setting a language other than English, please email support@cyotek.com
• Speed and estimated downtime time calculations were incorrect and could cause a crash when downloading large files (#466)

Removed

• The PDF meta data provider has been removed

The following changes are a sneak-peak of features that are currently being experimented with in separate branches from the core product. Due to this they are not available in nightly builds (and there is no guarantee they will be merged into the core branch).

Added

• Crawl state persistence (pause, resume) [#165]

Changed

• Crawling now uses breadth first when prioritising enqueued items