26 August 2020

Curl - a helping hand with redirects

This article was migrated from an older iteration of our website, and it could deviate in design and functionality.

Curl is a versatile tool and with a simple few commands it gives you easy insight into redirects along with the how and why of what’s happening along the way.

Estimated read time : 7 minutes

Introduction

In this blog post I will talk about the benefits of using a command line tool to view and troubleshoot website redirects and rewrites.

If command line tools is not for you, then you can go straight to the “Alternatives” section, but you might lose out on a perk or two from Curl.

What is Curl

To quote the the Curl website:

Curl is a command line tool for transferring data with URLs. It is an open source software used in multiple types of devices from cars to printers to television sets.

https://curl.haxx.se/

Benefits and use

As mentioned earlier, Curl can be used for multiple tasks, but today I will focus on how to use it as help when working with website redirects and rewrites.

When writing or troubleshooting website redirects and rewrites, one is often hindered by web browser caching. There is often need for small changes in the redirect or rewrite rules when working by trial and error and caching will make it impossible to see the effects of the changes.

By using a command line tool such as Curl there is no caching and therefore the results of every attempt and re-attempt will be displayed correctly.

Website projects often contains numerous redirects and rewrites.

They usually include:

a whitelist to exclude from any redirect or rewrite
converting HTTP to HTTPS
adding “www” to the URLs
adding trailing slash
converting URLs to lower case
redirect maps for legacy URLs

For SEO reasons the number of redirects and rewrites a page needs to make before reaching its destination should be as few as possible - in best case zero.

Curl helps by displaying each redirect or rewrite in the chain and makes it easier for you to see why a URL redirect leads astray.

Getting started

Head over to the Curl download page to download and install the package at the Curl download page. For this use you want the “curl executable” (and most likely the generic version).

Commands

A selection of useful commands when using Curl for redirects or rewrites:

-I --head, only displays headers making it useful for getting a minimalistic and easy to follow redirect chain.
-k, --insecure, by default Curl requires the connection to be secure but this option will ignore that. Perfect for when working with unsigned SSL certificates.
-L, --location, if the server responds with a redirect HTTP code, Curl will do a new request to that new URL. This is what enables us to use Curl for working with redirects and rewrites.

A full list of commands can be found in the Curl manual.

Examples

Simple HTTPS and WWW addition

>curl -I -L http://testwebsiteone.com

HTTP/1.1 301 Moved Permanently
Content-Length: 146
Content-Type: text/html; charset=UTF-8
Location: http://www.testwebsiteone.com/
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:01:21 GMT

HTTP/1.0 301 Moved Permanently
Location: https://www.testwebsiteone.com/
Server: BigIP
Connection: Keep-Alive
Content-Length: 0

HTTP/1.1 200 OK
Cache-Control: private,public
Content-Length: 79422
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/8.5
Set-Cookie: ASP.NET_SessionId=xxxccr1u2baoevjqeoq03mfq; path=/; HttpOnly; SameSite=Lax
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:01:21 GMT
Connection: close
Set-Cookie: sss_lb=3424072458.47873.0000; path=/; Httponly; Secure

Legacy URL

>curl -I -L http://testwebsitethree.fi/fi/paloauto/15-helikopteri-hevonen/

HTTP/1.1 301 Moved Permanently
Content-Length: 185
Content-Type: text/html; charset=UTF-8
Location: https://www.testwebsitethree.fi/fi/paloauto/15-helikopteri-hevonen/
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:19:32 GMT

HTTP/1.1 301 Moved Permanently
Content-Length: 182
Content-Type: text/html; charset=UTF-8
Location: https://www.testwebsitethree.fi/paloauto/15-helikopteri-hevonen/
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:19:33 GMT

HTTP/1.1 301 Moved Permanently
Content-Length: 188
Content-Type: text/html; charset=UTF-8
Location: https://www.testwebsitethree.fi/paloauto/15-helikopteri-paloauto/
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:19:33 GMT

HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 181592
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/10.0
Set-Cookie: ASP.NET_SessionId=2ilgjb3kc40vtfzpajhzztpz; path=/; HttpOnly
X-AspNetMvc-Version: 5.2
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:19:33 GMT

Redirect

>curl -I -L http://testwebsitetwo.com/de-ch/stadt/bananen-eis/dampfschiff-poker/

HTTP/1.0 301 Moved Permanently
Location: http://www.testwebsitetwo.com/de-ch/stadt/bananen-eis/dampfschiff-poker/
Server: BigIP
Connection: Keep-Alive
Content-Length: 0

HTTP/1.1 301 Moved Permanently
Date: Thu, 09 Jul 2020 13:04:46 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 203
Connection: close
Set-Cookie: __cfduid=dd26c2d0ead6c141ec6095c7a962adcb21594299886; expires=Sat, 08-Aug-20 13:04:46 GMT; path=/; domain=.testwebsitetwo.com; HttpOnly; SameSite=Lax
Location: https://www.testwebsitetwo.com/de-ch/stadt/bananen-eis/
Set-Cookie: TiPMix=56.149782778765; path=/; HttpOnly; Domain=www.testwebsitetwo.com; Max-Age=3600
Set-Cookie: x-ms-routing-name=self; path=/; HttpOnly; Domain=www.testwebsitetwo.com; Max-Age=3600
X-Powered-By: ASP.NET
Set-Cookie: ARRAffinity=a9629811ceb177436278d62fe4849d470cb174350ac4a8411f3ecc2a5d337d8c;Path=/;HttpOnly;Domain=www.testwebsitetwo.com
CF-Cache-Status: DYNAMIC
cf-request-id: 03d546c4f50000c76516a09200000001
Server: cloudflare
CF-RAY: 5b0240b4b879c765-AMS

HTTP/1.1 200 OK
Date: Thu, 09 Jul 2020 13:04:47 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Set-Cookie: __cfduid=d731d512c4dfc944c9be18d7aedeceb5d1594299887; expires=Sat, 08-Aug-20 13:04:47 GMT; path=/; domain=.testwebsitetwo.com; HttpOnly; SameSite=Lax
Cache-Control: private
Last-Modified: Fri, 22 May 2020 16:06:20 UTC
Vary: Accept-Encoding
Set-Cookie: TiPMix=27.0945826205865; path=/; HttpOnly; Domain=www.testwebsitetwo.com; Max-Age=3600
Set-Cookie: x-ms-routing-name=self; path=/; HttpOnly; Domain=www.testwebsitetwo.com; Max-Age=3600
Set-Cookie: ASP.NET_SessionId=dlzpeoqbsmbjutvhjuyamsf4; path=/; HttpOnly; SameSite=Lax
X-AspNetMvc-Version: 5.2
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-AspNet-Version: 4.0.30319
Request-Context: appId=cid-v1:09d56c47-bf06-48ab-beb5-4a7d4638f5f2
Access-Control-Expose-Headers: Request-Context
X-Powered-By: ASP.NET
Set-Cookie: ARRAffinity=bd312ecdcc0398778b4727ae09c806611d7c6e382b5247f6035067f542c76340;Path=/;HttpOnly;Domain=www.testwebsitetwo.com
CF-Cache-Status: DYNAMIC
cf-request-id: 03d546c63d0000fa80ef08e200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5b0240b6cd7efa80-AMS

Unsigned SSL certificate

>curl -I -L -k http://staging.testwebsitefour.com

HTTP/1.0 301 Moved Permanently
Location: https://staging.testwebsitefour.com/
Server: BigIP
Connection: Keep-Alive
Content-Length: 0

HTTP/1.1 302 Found
Cache-Control: private,public
Content-Length: 145
Content-Type: text/html; charset=utf-8
Location: /LoginRedirect?ReturnUrl=%2f
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:27:46 GMT
Connection: close

HTTP/1.1 302 Found
Cache-Control: private,public
Content-Length: 149
Content-Type: text/html; charset=utf-8
Location: /logins/episerver/?ReturnUrl=%2f
Server: Microsoft-IIS/8.5
Set-Cookie: ASP.NET_SessionId=1b32muw3tvpftnq0lt0ht0lr; path=/; HttpOnly; SameSite=Lax
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:27:46 GMT
Connection: close

HTTP/1.1 200 OK
Cache-Control: private,public
Content-Length: 9137
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/8.5
Set-Cookie: ASP.NET_SessionId=cfq115af1qh1jo0nwq2f0a10; path=/; HttpOnly; SameSite=Lax
X-Powered-By: ASP.NET
Date: Thu, 09 Jul 2020 13:27:46 GMT
Connection: close

Alternatives

Redirect Check

Redirect Check is a great website alternative tool if you do not want to install Curl on your computer or hassle with terminal commands.

One drawback is lack of support for locally hosted domains.

Redirect Check website.

Redirect Path

This handy tool is an easy to use Chrome Extension to get a visual representation of the redirect chain and its information for each step.

Redirect Path extension on Chrome web store.

Final words

Curl is a very versatile tool and its capabilities extends far beyond my example of using it as a redirect checker. With simple few commands it gives you easy insight into redirects along with the how and why of what’s happening along the way.

I use this tool on a weekly basis when fine tuning a redirect chain or troubleshooting a critical bug - it has helped me countless times with its cache-free responses and localhost support.

Hopefully this tool will help you too.

If you have any tool tips of your own, please share them in the comments.