Cyotek WebCopy Roadmap Copy websites locally for offline browsing

The following enhancements are ideas to be added to WebCopy over time. The features listed here are in no particular order nor are a guarantee that such a feature will be implemented.

You can also view upcoming changes.

• Add the ability to localize the product
• Support for detecting URL's within JavaScript code.
• Multi threaded crawling
• Ability to split a website in sections, allowing different settings to be applied to each, for example a forum might be marked as changes daily, while a FAQ area only monthly.
• Update the sitemap.org plugin to utilize the new last modified date support when generating sitemaps.
• For ASP.net sitemaps, the ability to enable security trimming (probably tied in with the split into sections enhancement)
• The ability to specify more advanced exclusion filters.
• Better query string support, such as the ability to remove values.
• Product help is a little weak in places
• Detect obsolete exclusions
• Add an online version
• Extend the reporting functionality to allow custom reports to be created
• Add the ability for content to be modified, such as removing HTML elements
• New rule support for crawl depth
• The ability to have multiple rules with the same base URI, and additional flags to control if a rule should be inherited
• The ability to control sorting via rules, currently all sorting is alphabetic by folder name or page title
• Add the ability to process rules based on sitemap type rather than a single global list
• Add the ability to allow content lengths to be used as a rule source, for example to prevent the downloading of files above a given size
• Rewrite list based inline editors to be less buggy
• Introduce a new chained filter rule engine, similar to Outlook Rules

• The ability to check files in and out of source code control Abandoned
• The ability to export and import robot.txt files Implemented in SMC 1.0.0.1
• Disable exclusions without having to delete them. Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.0
• The ability to map where page links have come from. Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.0
• Extend the plugin framework to allow more advanced providers to be added, and support for custom addons. Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.1
• The ability to store a hash and last modified date in the crawler project Implemented in SMC 1.0.0.1 / WebCopy 1.0.0.1
• Find/filter functionality for lists Implemented in SMC 1.0.0.3 / WebCopy 1.0.0.0
• The experimental filtering functionality added to the Link Map dialog makes it look cluttered. Improve the UI so the filtering can be extended. Implemented in SMC 1.0.0.3 / WebCopy 1.0.0.0
• More command line parameters to allow for fully automating the process. Implemented in SMC 1.0.0.4 / WebCopy 1.0.0.1
• Add support for sub domains. Implemented in SMC 1.0.0.6 / WebCopy 1.0.0.1
• Add support for 401 challenge requests. Implemented in SMC 1.0.0.6 / WebCopy 1.0.0.1
• Compression support for faster crawling. Implemented in SMC 1.0.0.6 / WebCopy 1.0.0.1
• Expand the options dialog to allow disabled plugins to be re-enabled, or in fact disable existing plugins if required. Implemented in SMC 1.0.0.7 / WebCopy 1.0.0.2
• The ability to upload sitemaps to your servers via FTP Implemented in SMC 1.0.0.10
• The ability to "ping" search engines when you update your sitemaps Implemented in SMC 1.0.1.0
• Better reporting for page errors, skipped pages, and external pages. Implemented in SMC 1.0.1.4
• A better product icon Implemented in SMC 1.0.0.1 / WebCopy 1.0.1.1
• Update the GUI to support some of the crawler settings not currently exposed such as supported content types. Implemented in SMC 1.0.1.0
• Expanding the new Link Map feature to be a diagram rather than a list, allowing for continuous navigation through the links whilst continuing to be able to see previous navigation. Implemented in SMC 1.0.1.0
• Convert the project format to binary for a smaller file size and faster processing Implemented in SMC 1.2.0.0 / WebCopy 1.0.2.0
• More intelligent exclusion processing so you can modify the exclusion list and dynamically update the sitemap without having to re-analyze the site. Implemented in SMC 1.0.4.0 / WebCopy 1.0.7.0
• Proxy server support Implemented in SMC 1.0.5.0 / WebCopy 1.0.9.0
• Speed limit support Implemented in SMC 1.0.5.0 / WebCopy 1.0.9.0