When crawling a page, it fails with a "method not allowed" error (code 405)

This error occurs when you try to perform an action on a given URI which is not allowed.

While the HTTP protocol defines a number of methods, WebCopy makes use of only three of these - HEAD, GET and POST.

Header Checking

By default, WebCopy issues the HEAD method before crawling any URI which provides important information such as the content type and length before actually trying to download any content. This speeds up crawls where you are excluding content types that belong to large binary files. However, if the web server doesn't support, or has disabled, the HEAD method, then any crawl of that server will fail.

If this happens, you need to disable the use of the HEAD command by WebCopy. To do this, display the Project Properties dialog, select the Advanced section, then uncheck Use Header Checking. Click OK to save your changes and close the dialog, then retry the crawl.

Posting a form

If you are posting a form at the start of a crawl, for example to authenticate using forms authentication, then you may get this error trying to post to a URI that doesn't support the POST method. Change the URI to point to the current location. WebCopy only supports posting forms using the POST method.

Leave a Comment

While we appreciate comments from our users, please follow our posting guidelines. Have you tried the Cyotek Forums for support from Cyotek and the community?

Styling with Markdown is supported