php - Speed up cURL page login and scraping -
this question has answer here:
i have function logins site , searches string in following page. process takes 10 seconds, wanted see if there speed up. wonder if possible have curl login persist on clients session or maybe search document better.
public function curllogin($url, $post_values, $cookiejar) { $timeout = 30; $curl_connection = curl_init(); curl_setopt($curl_connection, curlopt_url, $url); curl_setopt($curl_connection, curlopt_timeout, $timeout); curl_setopt($curl_connection, curlopt_useragent,"mozilla/4.0 (compatible; msie 6.0; windows nt 5.1)"); curl_setopt($curl_connection, curlopt_cookiejar, $cookiejar); curl_setopt($curl_connection, curlopt_cookiefile, $cookiejar); curl_setopt($curl_connection, curlopt_cookiesession, 0); curl_setopt($curl_connection, curlopt_header, 1); curl_setopt($curl_connection, curlopt_returntransfer, 1); curl_setopt($curl_connection, curlopt_ssl_verifypeer, 0); curl_setopt($curl_connection, curlopt_post, 1); curl_setopt($curl_connection, curlopt_postfields, $post_values); curl_setopt($curl_connection, curlopt_httpheader, array("content-type: application/x-www-form-urlencoded")); curl_exec($curl_connection); return $curl_connection; } public function curlpost($curl_connection, $url, $post_values, $cookiejar) { $timeout = 30; curl_setopt($curl_connection, curlopt_url, $url); curl_setopt($curl_connection, curlopt_timeout, $timeout); curl_setopt($curl_connection, curlopt_useragent,"mozilla/4.0 (compatible; msie 6.0; windows nt 5.1)"); curl_setopt($curl_connection, curlopt_cookiejar, $cookiejar); curl_setopt($curl_connection, curlopt_cookiefile, $cookiejar); curl_setopt($curl_connection, curlopt_cookiesession, 0); curl_setopt($curl_connection, curlopt_header, 1); curl_setopt($curl_connection, curlopt_returntransfer, 1); curl_setopt($curl_connection, curlopt_ssl_verifypeer, 0); curl_setopt($curl_connection, curlopt_post, 1); curl_setopt($curl_connection, curlopt_postfields, $post_values); curl_setopt($curl_connection, curlopt_httpheader, array("content-type: application/x-www-form-urlencoded")); $result = curl_exec($curl_connection); return $result; } $cookiejar = tempnam ("/tmp", "curlcookie"); $curl_connection = $this->curllogin($login_url, $post_values, $cookiejar); $result = $this->curlpost($curl_connection, $next_url, $params, $cookiejar); if (strpos($result,'string 1') > 0) { $success = true; $message = 'string 1 present'; }else if (strpos($result,'string 2') > 0){ $success = false; $message = 'string 2 present'; }else if (strpos($result,'string 3') > 0){ $success = false; $message = 'string 3 present'; }else{ $success = false; $message = 'none of above strings present.'; } curl_close($curl_connection); unlink($cookiejar);
you can avoid logging in every time reusing cookiejar.
create file called cookies.txt in directory containing script , assign: $cookiejar = 'cookies.txt'.
after running script first time, remove call curllogin() function , curlpost() function should use cookies correctly , return data if logged in.
remember, curlopt_cookiefile specify "read" cookies , curlopt_cookiejar want response cookies written.
so without curlopt_cookiejar in curlpost() function.
Comments
Post a Comment