Products Blog Articles and information on Cyotek's products

This is quite a long post so I'm just going to add an important bit of news here - WebCopy 1.1 will not be able to be installed or ran on Windows XP

WebCopy, like most Cyotek products, is built in C# using Microsoft .NET Framework 3.5, thus allowing it to run on Windows XP onwards. Each time the product is built, a batch file is manually ran which goes away and compiles the solution, signs the files, does some "deployment ready" checks, generates the documentation and then generates the setup. Tests are not run as part of this process as generally they are always running in the IDE via NCrunch.

Once a build has been completed and we're thinking about deploying it, we then fire up a Windows XP virtual machine in VirtualBox and proceed to run a basic smoke test - this usually involves opening the demo project, downloading the demo website, then creating a new project, pointing it to a small 3rd party website, downloading that, then saving the project. Just the basics in other words.

If this smoke test passes, then we upload the setup to cyotek.com, copy the Upcoming Changes content into a new document, then link it all together - and a new release is ready for end users to download.

It's actually not a laborious process (except for the smoke test) as for the most part, while not fully automatic, all the major actions are handled by several batch scripts and specialist actions. But it would be very nice to have this all done automatically.

Which is why I feel an intense surge of glee whenever I look at this display:

This is TeamCity, the product we are using for continuous integration- basically, whenever changes are checked into version control, TeamCity goes away and creates the build without us having to manually trigger it. It also gathers the build artefacts ready for more processing.

I evaluated both Jenkins and TeamCity for this job, starting with Jenkins. It didn't require any configuration changes on the server or agent hosts, and produced a build. (Eventually anyway, it took a lot of effort to get the WebCopy build into a shape where it would work purely via checkouts. Un-versioned files and hard coded tool paths do not help!). But, I didn't warm to Jenkins, mainly because it's so incredibly ugly, and because out of the box functionality is quite limited (for .NET shops) unless you want to start installing a multitude of plugins.

Next I tried TeamCity. I originally looked at this back in 2013 (with version 7) and ended up shelving it then (mostly because the builds just wouldn't run correctly). While Jenkins worked out the box, TeamCity needs a full blown database server of some sort (SQL Server for us!) and need tinkering to the firewalls on both the server and the agent. But in terms of UI user friendliness it's light years ahead of Jenkins. And as the screenshot shows, WebCopy is happily being built.

But what does this mean?

Firstly, more work needs to be done with the build process - the batch files are sometimes a bit fragile and don't make for easy error detection. I'd been looking at FAKE a couple of weeks back as a possible replacement, but there are other options such as NANT and probably a bucketful of others. Once the build has been converted to use something else, then I'll start adding the automated unit/integration tests into the process.

And then... I plan on having TeamCity talk to cyotek.com to automatically publish "nightly" (except they won't be that frequent!) builds to hopefully get new versions of the software out quicker.

We've also been spending substantial time recently rewriting core tests for WebCopy's crawling engine so that we can be much more confident in it's abilities, know immediately if regressions are introduced, and to ensure oversights and bugs are swiftly rectified. Oh yes, and to make the things faster.

Some parts of WebCopy are crying out for performance improvements. Lets start with the basics, such as threading. Threading is a way of a program doing more than one thing at once. WebCopy does use threads, but only to keep the UI from being blocked, and for supporting actions such as update checks or RSS updates. The core crawling is done on a separate thread, but essentially all ancillary actions are a big chain, one link after another.

While there's nothing really stopping use from staying with .NET 3.5 and expanding the threading capacity, newer versions of .NET make this much easier with lots of built in goodness like concurrent collections, parallel workers and much more easy ways of making asynchronous calls - the ones in WebCopy currently are difficult to debug when things go wrong.

.NET 4.5 is built into Windows 8 onwards, and is available as a download for Vista and Windows 7. It is not available for Windows XP however, so it is highly likely that WebCopy 1.0.10.1 will be the last version available for this OS.

Based on submitted analytic's data, the breakdown of WebCopy users per operating system looks like this

At 12% the XP user base is larger than I would have expected, but I think it's time to drop support regardless.

• 2015-04-11 - First published
• 2020-11-23 - Updated formatting

While we appreciate comments from our users, please follow our posting guidelines. Have you tried the Cyotek Forums for support from Cyotek and the community?