The other day I wanted to automate some downloading from a username and password secured website. I wrote a quick script and it is working like a dream, below is the CURL part of the code that does the logging in and download.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
$username = 'myuser'; $password = 'mypass'; $loginUrl = 'http://www.example.com/login/'; //init curl $ch = curl_init(); //Set the URL to work with curl_setopt($ch, CURLOPT_URL, $loginUrl); // ENABLE HTTP POST curl_setopt($ch, CURLOPT_POST, 1); //Set the post parameters curl_setopt($ch, CURLOPT_POSTFIELDS, 'user='.$username.'&pass='.$password); //Handle cookies for the login curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt'); //Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL //not to print out the results of its query. //Instead, it will return the results as a string return value //from curl_exec() instead of the usual true/false. curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //execute the request (the login) $store = curl_exec($ch); //the login is now done and you can continue to get the //protected content. //set the URL to the protected file curl_setopt($ch, CURLOPT_URL, 'http://www.example.com/protected/download.zip'); //execute the request $content = curl_exec($ch); //save the data to disk file_put_contents('~/download.zip', $content); |
How can I get cookie.txt?
cookie.txt is created by cURL and used to store the login session info
i need this for my android application to login in craigslist.org and post through my app,
i tried same to authenticate into craigslist.org but it is not working 🙁
can you please suggest me what to do..
You need to look at the websites source and see if you are passing al the required info and if the username and password fields have the same name as in the example.
Hadn’t used curl before and was on my third day of trying web tweaks. Finally was about to give up and tried one last one, this one. It worked the first time. Perfect!
Why didn’t you try this one first? 😛
Thank you Andy, it help me a lot, now I can get the content after the login but only if the url points directly to a url that is a file. I want to download a file that is created after sendime some parameters along with the URL like http://myURL.com/admin/history/xls?o=&from_hours=00:00&to_hours=23:59. I have tried to use these parameters in the same way as in the login step with no success. The URL is fine as I tested it in the browser and I receive the file, The cookie file is being created too. Any ideas. Joao Carlos
I would say have a look at the PHP manual page for curl_setopt, I haven’t seen your code but you might need to tell curl to use GET after the login stage using CURLOPT_HTTPGET. Hope it helps.
One issue I am having is when the input names are something like this : ctl00$bodyContent$email,ctl00$bodyContent$pass
The validation doesnt happen correctly. Even if I use http_build_query() around the array that holds these names and values.
Any ideas?
Getting error Notice: Undefined index: HTTP_ACCEPT_ENCODING
Any help regarding this will be appriciated
it’s really helpful for me.thank u.
In my case cookie.txt is never created. The problem is , Am able to login the site but if i click any link on that site it will show me 404 error. Any clue matey ?
Hi,
I am new to curl, I have been given a task to login into a website and retrieve a line of data.
I copied your codes and changed the parameters but nothing displays when I run the script. is there a way to check what is wrong? I did if( curl_exec($ch)) echo 1; else echo 0; and I kept getting 0.
Kindly help.
You don’t say if this curl_exec() is for the log in or the download step? Also var_dump() the output of curl_exec() as it might give you a clue. Also please check that the login form username and password field names match the ones on the ‘Set the post parameters’ line.
Thanks a ton dude. This is so helpful!!
Andy i ..need a help..regarding getting into the website without logging in..it is on apache server
so how can i do it.
Can’t be done, you need to log in with a username and password
I copied your code, but it returns nothing.. is it anything to do with the apache server’s version? is there any files that I have to downloaded first, before I could use this curl? thanks before..
Nice one, thanks!
It got me on the right track. Strangely I can only get it to work if i put:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1) ;
after the login ($store = curl_exec($ch);)
If it works it works, but would you happen to know why this could be? Is it a cookie thang?
I’m new to PHP and curl. Two things puzzle me about line 15.
1) what is the meaning of the & in &pass, why isn’t it $pass?
2)Why is there no separator between $username and &pass? if username=joe then you get $user=joe&pass
(unless the & serves as a separator)
Any way I’m confused by line 15 of the code. Can someone try to explain it to me?
This line is setting the HTTP POST fields, its setting up the username and password to pass to the server.
These are in the format of a query string. See http://en.wikipedia.org/wiki/Query_string for more info on query strings
dizzyman, see here once i will explain it to you.
user=sailesh&pass=sailesh65 <<<<<< this is the executing code.
And there we are creating similer to this line.
$username = sailesh
$password = sailesh65
so 'user='.$username.'&pass='.$password = user=sailesh&pass=sailesh65
and this . will add all string there.
i tried my best to explain it.
hope you understood it.
me too started learning php just 4 days ago
I cannot seem to get this to work. Can someone help me out. It just doesn’t do anything. All I see is a blank white screen and no cookies.txt gets created. Also no output exists when I echo the curl_exec statement. How do I debug this or any idea what’s wrong?
create cookies.txt manually
Thanks Andy! Really helpful for me!
Hi Andy! Great code. How can I make it work if the website is also generating a security token every time you access the login url to send it on the post? Thanks again!
You’ll probably need to do another curl call. Set the cookie and then pull the login page. Use PHP to get the token out of the HTML and then add it to the login post parameters on the next curl call.
Hi Andy, thanks for the fast reply. I just tried that and what it gets back is that my session expired and I need to login again 🙂 I did a file_get_contents on the login page, parse the token and send it on the login page post the securityToken together on the post vars.
Don’t use file_get_contents(), you need to use curl to get the page as it will store the cookie with your session id in it.
Something like this:
$username = ‘myuser’;
$password = ‘mypass’;
$loginUrl = ‘http://www.example.com/login/’;
//init curl
$ch = curl_init();
//Set the URL to work with
curl_setopt($ch, CURLOPT_URL, $loginUrl);
//Handle cookies for the login
curl_setopt($ch, CURLOPT_COOKIEJAR, ‘cookie.txt’);
//execute the request
$loginPage = curl_exec($ch);
//Now extract the token from the html
$token = myGetTokenCode($loginPage);
// ENABLE HTTP POST
curl_setopt($ch, CURLOPT_POST, 1);
//Set the post parameters
curl_setopt($ch, CURLOPT_POSTFIELDS, ‘user=’.$username.’&pass=’.$password.’&token=’.$token);
//Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
//not to print out the results of its query.
//Instead, it will return the results as a string return value
//from curl_exec() instead of the usual true/false.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//execute the request (the login)
$store = curl_exec($ch);
//the login is now done and you can continue to get the
//protected content.
//set the URL to the protected file
curl_setopt($ch, CURLOPT_URL, ‘http://www.example.com/protected/download.zip’);
//execute the request
$content = curl_exec($ch);
//save the data to disk
file_put_contents(‘~/download.zip’, $content);
Hi Andy, thanks for replying on a Saturday. How can I send you the file I’m using, I think I put it how you advised but still no good for me. Thank you.
Andy! Progress!!!! I was able to login, the only difference was I moved the line curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); to go after the first curl call to get the cookie, it looks like when print the page it reset the cookies! Thanks for all your help! You are a star!
Happy to help.
Can you use this cURL call to login and create a session on a website on another domain? So that if you click on a link directing you to that webpage you’re already logged in?
Can you use this cURL call to login and create a session on a website on another domain?
– Yes
So that if you click on a link directing you to that webpage you’re already logged in?
– Assuming I am understanding what you are trying to do correctly, then no (or maybe depending on lots of things).
Hello,
I try the code, but without success. I think I need the same code as KItara, login to OpenCart with &token and then to next URL. Can someone write working code for this?
I am using this in Cpanel authentication:
Here is my code:
$loginUrl = ‘http://69.16.197.227:2086/login’;
$user=”myusername”;
$password=”mypassword”;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $loginUrl);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, ‘user=’.$user.’&pass=’.$password);
curl_setopt($ch, CURLOPT_COOKIEJAR, ‘cookie.txt’);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$store = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, $url);
$content = curl_exec($ch);
curl_close($ch);
But it always return Access Denied, please advise i am doing wrong?
Have a look at my comments from the 11th January and try that.
Hi Andy,
Question, I want to attempt to use curl to logon to my icloud account. I’ve looked at the login page at http://www.icloud.com and cannot workout what the username and passwords fields are that are used by icloud by viewing source. Any chance of a pointer?
Thx
Ok first pointer is use the developer console to view the source, in chrome right click on the input field and then click inspect element. This will open up developer tools with the source at the point of the element.
Second pointer is that looking at the element you can see that the name of the username input field is something random like sc1804. If you refresh and repeat pointer one you will see that the field name has changed. This is a problem but probably not impossible to work around.
Look at my connects from 11th January, you will have to do a similar thing but instead of getting the token you will have to parse the HTML and get the username and password field names. Then you can do the login POST.
Good luck!
Thanks for this helpful post
Can i use this for Caspio bridge app login?
I’m not familiar with caspio bridge but if it’s a web app then probably.
Hi Andy
One I want to do 2 things Using curl
1) Login on olx site (http://assam.olx.in/login.php/)
2) fill the Posting form (http://www.olx.com.br/posting.php?categ_id=367)
please help me about that issue i try that code for login
but olx.in is allow me to get login
thanks
Hi Arvind, did your login at olx worked? could you share the script?
I am trying to use curl to setup some links that login to a password protected directory and shows some pages. I have the following code:
$username=’myusername’;
$password=’mypassword’;
$url=’www.mywebsite.com/mypage.html’;
//init curl
$ch = curl_init();
//set the URL to the protected file
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_USERPWD, $username . ‘:’ . $password);
//execute the request
curl_exec($ch);
It seems to work as it gets into the password protected directory but the page it shows contains iframes, each iframe (there are 4) shows page not found? It must be hitting the page in order to see the iframe layout so why the 404’s? Any ideas.
Just to note: for some of you obviously not the individuals needing tokens etc but a few above posts may have items cached hence the reason for the errors even though you followed Andy’s great code. Make sure to close the curl afterwards so it doesn’t appear you have an error once you’ve successfully updated it. I recommend an error catch to automatically close the curl should it not run correctly.
Hi, I just used your code and it seems easy to understand but when I do implement I got an error after debugging: This page should automatically redirect. If nothing is happening please use the continue link below.
Continue
Then I checked whether the cookies are enabled or not and browser cookies are enabled. Can you please help me for this.
You need to use curl to retrieve the next page as that’s what has the authenticated cookie
Yes I am using curl but I got the above error when I debugged and then I click manually on the continue link then I got issue with the cookies. So I checked whether they are enabled or not and they are.
You can’t click on it manually because it is curl that has the authorised cookie not your browser. This code is not really meant for the browser
Hi Andy
I want login to a forum based on xenforo and click on a link every 2 minutes by cron jobs in Cpanel. To do this, what should I change in code?
thank you
Hello Andy
Please check my code below:
$username=”#########”;
$password=”##########”;
$loginUrl = ‘https://accounts.craigslist.org/login/’;
//init curl
$ch = curl_init();
//Set the URL to work with
curl_setopt($ch, CURLOPT_URL, $loginUrl);
// ENABLE HTTP POST
curl_setopt($ch, CURLOPT_POST, 1);
//Set the post parameters
curl_setopt($ch, CURLOPT_POSTFIELDS, ‘inputEmailHandle=’.$username.’&inputPassword=’.$password);
//Handle cookies for the login
curl_setopt($ch, CURLOPT_COOKIEJAR, ‘cookie.txt’);
//Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
//not to print out the results of its query.
//Instead, it will return the results as a string return value
//from curl_exec() instead of the usual true/false.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//execute the request (the login)
$store = curl_exec($ch);
//the login is now done and you can continue to get the
//protected content.
//set the URL to the protected file
curl_setopt($ch, CURLOPT_URL, ‘https://accounts.craigslist.org/login/home’);
//execute the request
$content = curl_exec($ch);
curl_close($ch);
Here I am trying to log in into my craigslist account and redirected to my home page. But its not working.
Can you please help me regarding this.
hi, dear i want to parse this site info…. i am new to parsing
http://182.180.76.176/login.aspx
tell me how it will work… it show false
hi, dear i am new to parsing .. i want to parse contents of this web
http://182.180.76.176/login.aspx with guest both user name and passeotd when i use this code it redirect to error page … how t will done… thanks
hi andy, i am trying to use a free web hosting service and run your code to always stay logged into a website(plug.dj) i dont know if its even possible to do but its not working. if it is possible could you set me on the right track please thanks.
Andy, thank you. You made it so I could understand the code. That helps tremendously.
not worked for me . I don’t know how cookie.txt fill will generate and how will it login to the website??
Hello Andy,
I stucked with my code. I am trying to fetch the user details from Twitter through cURL request. But it is saying the error as ‘403 Forbidden: The server understood the request, but is refusing to fulfill it’. Please help me on this.
$ch = curl_init();
$sTarget = “https://mobile.twitter.com/session/new”;
curl_setopt($ch, CURLOPT_URL, $sTarget);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_USERAGENT, “Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0”);
curl_setopt($ch, CURLOPT_COOKIEFILE, “/tmp/cookie.txt”);
curl_setopt($ch, CURLOPT_REFERER, $sTarget);
$html = curl_exec($ch);
curl_close($ch);
$_regex = ‘/<meta name="csrf_id" content="([^"]*)"/';
preg_match($_regex, $html, $output);
$authenticity_token = $output[1].'’;
$username = ‘#######’;
$password = ‘#######’;
$loginUrl = ‘https://mobile.twitter.com/sessions’;
//init curl
$ch1 = curl_init();
//Set the URL to work with
curl_setopt($ch1, CURLOPT_URL, $loginUrl);
//Handle cookies for the login
curl_setopt($ch1, CURLOPT_COOKIEJAR, ‘cookie.txt’);
//execute the request
$loginPage = curl_exec($ch1);
// ENABLE HTTP POST
curl_setopt($ch1, CURLOPT_POST, 1);
$sPost = “session[username_or_email]=”.$username.”&session[password]=”.$password.”&return_to_ssl=true&wfa=1&remember_me=1&authenticity_token=”.$authenticity_token;
//Set the post parameters
curl_setopt($ch1, CURLOPT_POSTFIELDS, $sPost);
//Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
//not to print out the results of its query.
//Instead, it will return the results as a string return value
//from curl_exec() instead of the usual true/false.
curl_setopt($ch1, CURLOPT_RETURNTRANSFER, 1);
//execute the request (the login)
$store = curl_exec($ch1);
//the login is now done and you can continue to get the
//protected content.
//set the URL to the protected file
curl_setopt($ch1, CURLOPT_URL, ‘https://mobile.twitter.com/account’);
//execute the request
$content = curl_exec($ch1);
print_r($content);
curl_close($ch1);
Hi, sorry I cannot spare the time to help you work through this but I have approved the comment in the hope that someone else may help you.
Hello Andy,
Can you at least tell me where I made the mistake in code?
I cannot know that unless I spend the time looking into it, sorry
OK..Thanks for your time.!
its still failed for
when I print $content, it shows :
301 Moved Permanently
why?
I mean : its still failed for me
Is there any way to bypass captcha codes.
I am trying to fetch some details from a site that has captcha code.
How i bypass with cURL ? Or any other way so i can show the same captcha on my site and post to original site with other fields ?
How can I login into a page and anfter login be redirected to the website autheticated ?
That is not what this script is for, in this script any interaction with the remote site would need to be via curl, not the browser.
Hey Andy tried your code, It’s working for me but there is a small issue.
I have to login every time when I need the protected content. Is there any workaround for that since it will be very messy ‘cuz I need to parse a lot of pages.
If you keep the curl session open (don’t do curl_close() or the script ending) you should be able to make more curl calls to protected resources.
Hi Andy Thanks for sharing but how can we determine login is success full. I have to login and download the spreadsheet automatically. So is there any change in this script.
hi andy thnx for sharing
can you help me bro with this site http://www.indianrail.gov.in/pnr_Enq.html
i want to make code that send the parameter pnr number and want to get the status of the booked ticket unfortunately i tried alot with curl but failed can you help me to reach the code
View the source of the rail page and make sure that you are sending all the hidden fields as well as the correct names and values for all the form fields.
hi Andy, at the moment i have a script that could login using some post filelds or using a cookie that is saved when i log in via post. OK, but the problem is when i want to redirect to the target website, i shoud be loggen in, but i
m not. im just using header(‘Location: website.com’); before the closing the curl session. How do i pass the cookie when i redirect ? or is there another way to keep the session alive ? ThanksHi Spidey, this script is about having the script log in not a browser. The script can communicate with the site as a logged in user so long as it is all through curl commands and the cookie is used.
When you are trying to redirect your browser, your browser does not have the cookie that curl created and is using, and that is why the browser session isn’t logged in.
Hi, can this be used to login to students platform with username and password and then check somehow if something changed on the page. In my case I just want to get notified when some fields are changed on the page. Since the page doesn’t change often. If anything changes on the page.
Is this possible with the script?
Thanks!
Yes, you would just grab the page in question using the code on line 33 of the code in this post. Then you could process that to check for changes.
[…] I had same question and I found this answer on this website. […]
[…] “Using PHP and CURL to log in to a website” by Andy Hunt. […]
how to use GET method in curl??
Hi Andy; i want login to https website. i have given cert path in CURLOPT_CAINFO. and aslo save .pem file and have given path in php ini file . everything is on my server . but still it gives error “SSL certificate problem: unable to get local issuer certificate”
Is it an option to just ignore the SSL issue using the below options?
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
PHP Parse error: syntax error, unexpected ‘memory_limit’ (T_STRING) in /home/ateam/public_html/forse.php on line 33
i have that error when execute file
That is not an error with the code I have provided on this page. It’s an error in your code. Check line 33, you are probably doing something wrong.
My Hero Academia: Who Are The Top 10 Pro-Heroes?