Using cURL and having to store cookies between requests but don’t want to save them to file? Here’s how to keep them purely in memory.

How to cURL in Style with Cookies

If you’re reading this article, then chances are you are already using Bash, know curl enough to be dangerous, and use it on a regular base.

Or, StackOverflow just didn’t have the answer you were looking for and we happened to be near the top on your search engine of choice.

Either way, you are curling a website that first hands you a cookie in response to you signing in. The catch is that the website then expects to find that cookie in all subsequent requests. Or it will refuse to do anything at all.

Your problem now is how to persist that cookie between requests without exposing it.

A Popular But Insecure Solution

A popular solution is to store the cookie in a temporary file. What could possibly go wrong…?!

In the initial request below to the Ghost API on How Hard Can It Be?!, the returned cookie is stored in cookie.txt after signing in with username and password

curl -c cookie.txt \
     -d username="not-a-real-username" \
     -d password="not-a-real-password" \
     -H "Origin: localhost" \
     "https://www.how-hard-can-it.be/ghost/api/v2/admin/session/"

Side note: As you are sending username and password in plain text, make sure you are using HTTPS! Oh, and: Please don’t try these values on our hard working server — they are obviously for demonstration purposes only. Thank you!

With the cookie stored in cookie.txt, all subsequent calls to curl then simply reference the file as in

curl -b cookie.txt \
     -H "Content-Type: application/json" \
     -H "Origin: localhost" \
     "https://www.how-hard-can-it.be/ghost/api/v2/admin/posts/"

However, there’s a problem. And it’s the obvious one!

The cookie.txt file is persistently stored on disk.

This means that even after your curl commands have finished, access to the server is still granted via cookie.txt. And if you are using session cookies then access is actually granted indefinitely as the curl documentation states.

When curl writes cookies to this file, it will save all known cookies including those that are session cookies (without a given lifetime). curl itself has no notion of a session and it does not know when a session ends so it will not flush session cookies unless you tell it to.

Not good.

Deep Inside the cURL Documentation

There has to be a better solution. And there is. A popular StackOverflow post on the subject already has all the right pointers in it.

In the end, it’s a combination of temporary variables and the right curl input parameters that do not seem to be clearly listed in the user documentation on cookies.

The real magic regarding the output of cookies is buried deep inside the curl documentation of the CURLOPT_COOKIEJAR parameter where it says

Specify “-“ as filename to instead have the cookies written to stdout.

And the best news about this is that there’s a matching counterpart regarding inputs for cookies as well on the documentation of the CURLOPT_COOKIEFILE parameter that says

If you tell libcurl the file name is “-“ (just a single minus sign), libcurl will instead read from stdin.

Now, these two findings can be combined to keep cookies purely in memory!

A Purely Memory Based Solution

It’s fairly easy to extend the popular but insecure solution above.

The main change to the first curl command is to leverage the - option for the curl parameter -c. With the cookie now being written to stdout, all there remains to do is capture the cookie in the temporary variable cookie as in

cookie=$(curl -c - \
              -d username="not-a-real-username" \
              -d password="not-a-real-password" \
              -H "Origin: localhost" \
              "https://www.how-hard-can-it.be/ghost/api/v2/admin/session/")

The only change to the second curl command is to again leverage the - option for the curl parameter -b and read the cookie from stdin. When piping in the contents of ${cookie}, the entire command can be written as

echo "${cookie}" | curl -b - \
                        -H "Content-Type: application/json" \
                        -H "Origin: localhost" \
                        "https://www.how-hard-can-it.be/ghost/api/v2/admin/posts/"

Combining the two commands into a single Bash script then results in the following GitHub Gist

So, How Do You cURL?!

While the above works for me when it comes to keeping cookies purely in memory between requests when using curl, you may have an alternative or better way.

Think this is all rubbish, incomplete, massively overcomplicated, or simply the wrong tool for the job?! Feel free to leave a comment on the GitHub Gist or reach out to me on LinkedIn and teach me something new!

As always, prove me wrong and I’ll buy you a pint!