Scattered Thoughts

At work I was using a simple script that connected to a secure API, logged in and saved the result. This seems simple enough and is fast enough when calling it a few times, however one of the tasks I was then assigned was actually running this script for different values over 60,000 times! Each call (including the logging in) would take a little over a second, so calling it 60,000 times would take a while.

It was then decided to try and improve this by making the script only login once and save the session, so that subsequent calls would not have to login each time. This is one of the solutions we came up with ( because finding this online was not very easy )

The first step is the login, the agent variable is being set to trick the API into thinking that the request has come from a browser.

//Set the cookie file name
$cookiefile = tempnam("/tmp", "cookies");
$agent     = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
//Set the user name and password values
$USER      = 'user';
$PASS      = 'pass';
//The API url, to do the login
$url = "https://www.example.com/";

//Initialise CURL
$ch = curl_init();
//Set all the various options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_POST, 0); // set POST method
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
//Set the cookie file you want to use
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
curl_setopt($ch, CURLOPT_USERPWD, $USER.":".$PASS);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);

//execute the CURL
$mRes = curl_exec($ch);

Remember not to close curl at this point. Now we look at the loop that will carry out the 60,000 calls to the API

while ($row = [get row value for the 60,000 entries])
{
    $url       = “https://www.example.com/API?check=$row”;
    //We do not have to initialise CURL again
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_USERAGENT, $agent);
    curl_setopt($ch, CURLOPT_POST, 0); // set POST method
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    //Remember to use the same cookiefile as above
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
    //execute the CURL call
    $mRes = curl_exec($ch);
    //DEAL WITH THE RESULT FROM THE CURL CALL
}
//only close CURL once the work is finished.
curl_close($ch);

If you’d like to know more the various CURL options I have used above like ‘CURLOPT_SSL_VERIFYHOST’ etc, have a read through the curl_setopt page on php.net. You can also read more about CURL there as well.