The following enhancements are ideas to be added to WebCopy over time. The features listed here are in no particular order nor are a guarantee that such a feature will be implemented.

You can also view upcoming changes.

Pending Features

  • Add the ability to localize the product
  • Support for detecting URL's within JavaScript code.
  • Multi threaded crawling
  • Ability to split a website in sections, allowing different settings to be applied to each, for example a forum might be marked as changes daily, while a FAQ area only monthly.
  • Update the sitemap.org plugin to utilize the new last modified date support when generating sitemaps.
  • For ASP.net sitemaps, the ability to enable security trimming (probably tied in with the split into sections enhancement)
  • The ability to specify more advanced exclusion filters.
  • Better query string support, such as the ability to remove values.
  • Product help is a little weak in places
  • Detect obsolete exclusions
  • Add an online version
  • Extend the reporting functionality to allow custom reports to be created
  • Add the ability for content to be modified, such as removing HTML elements
  • New rule support for crawl depth
  • The ability to have multiple rules with the same base URI, and additional flags to control if a rule should be inherited
  • The ability to control sorting via rules, currently all sorting is alphabetic by folder name or page title
  • Add the ability to process rules based on sitemap type rather than a single global list
  • Add the ability to allow content lengths to be used as a rule source, for example to prevent the downloading of files above a given size
  • Rewrite list based inline editors to be less buggy
  • Introduce a new chained filter rule engine, similar to Outlook Rules

Completed Features

  • The ability to check files in and out of source code control Abandoned
  • The ability to export and import robot.txt files Implemented in SMC 1.0.0.1
  • Disable exclusions without having to delete them. Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.0
  • The ability to map where page links have come from. Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.0
  • Extend the plugin framework to allow more advanced providers to be added, and support for custom addons. Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.1
  • The ability to store a hash and last modified date in the crawler project Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.1
  • Find/filter functionality for lists Implemented in SMC 1.0.0.3 / WebCopy 1.0.0.0
  • The experimental filtering functionality added to the Link Map dialog makes it look cluttered. Improve the UI so the filtering can be extended. Implemented in SMC 1.0.0.3 / WebCopy 1.0.0.0
  • More command line parameters to allow for fully automating the process. Implemented in SMC 1.0.0.4 / WebCopy 1.0.0.1
  • Add support for sub domains. Implemented in SMC 1.0.0.6 / WebCopy 1.0.0.1
  • Add support for 401 challenge requests. Implemented in SMC 1.0.0.6 / WebCopy 1.0.0.1
  • Compression support for faster crawling. Implemented in SMC 1.0.0.6 / WebCopy 1.0.0.1
  • Expand the options dialog to allow disabled plugins to be re-enabled, or in fact disable existing plugins if required. Implemented in SMC 1.0.0.7 / WebCopy 1.0.0.2
  • The ability to upload sitemaps to your servers via FTP Implemented in SMC 1.0.0.10
  • The ability to "ping" search engines when you update your sitemaps Implemented in SMC 1.0.1.0
  • Better reporting for page errors, skipped pages, and external pages. Implemented in SMC 1.0.1.4
  • A better product icon Implemented in SMC 1.0.0.1 / WebCopy 1.0.1.1
  • Update the GUI to support some of the crawler settings not currently exposed such as supported content types. Implemented in SMC 1.0.1.0
  • Expanding the new Link Map feature to be a diagram rather than a list, allowing for continuous navigation through the links whilst continuing to be able to see previous navigation. Implemented in SMC 1.0.1.0
  • Convert the project format to binary for a smaller file size and faster processing Implemented in SMC 1.2.0.0 / WebCopy 1.0.2.0
  • More intelligent exclusion processing so you can modify the exclusion list and dynamically update the sitemap without having to re-analyze the site. Implemented in SMC 1.0.4.0 / WebCopy 1.0.7.0
  • Proxy server support Implemented in SMC 1.0.5.0 / WebCopy 1.0.9.0
  • Speed limit support Implemented in SMC 1.0.5.0 / WebCopy 1.0.9.0

Download

Download current and archived versions of Cyotek WebCopy

Download

Minimum Requirements

Donate

Donate