Logging into this forum using Python

[ivnja]

Hi all. After several years of not doing much coding, I'm trying to write a short program in Python where the first step is logging into this forum. I know my way around Python in general reasonably well, but have never tried writing code to post anything (such as login data) vs scraping info (generally, neatly organized data with clear headers) from servers. Google tells me that the "requests" library is appropriate for this, and below is what requests's documentation tells me I should be doing, with the print statement there to show me the html for the resulting page:

Code: Select all

import requests

payload = {'username': 'ivnja', 'password': 'hunter2'}
r = requests.post('http://www.forums.xkcd.com/ucp.php', data=payload)
print(r.text)


What this should be doing is logging me into the forum and dropping me at my control panel, but as far as I can tell the login is failing and I end up at the same "Please login" page (i.e. this:

Spoiler:

). I've tried running this with both http://www.forums.xkcd.com/ucp.php and http://www.forums.xkcd.com/ucp.php?mode=login as the URLs. One thing I did notice is that if, while signed out, I go to forums.xkcd.com in my browser and click the login link, it sometimes brings me to the same page but with an sid appended after ?mode=login, but sometimes does not. Copying one of the sid-appended urls into my Python code and running it with that did not work, either.

The thing is, it's not showing any error that I can see. When I attempt to log in with an incorrect password in my browser, red text saying "You have specified an incorrect password. Please check your password and try again. If you continue to have problems please contact the Board Administrator" shows up, and in the page source you can find the following:

Code: Select all

      <div class="error">You have specified an incorrect password. Please check your password and try again. If you continue to have problems please contact the <a href="./memberlist.php?mode=contactadmin">Board Administrator</a>.</div>

Ditto for an incorrect username. When I run my code, I don't get that in the output no matter what I put for username or password, leading me to believe that the Python code isn't actually managing to post anything to attempt the login.

Would anyone who is familiar with the requests library (or has a different way of doing it) mind guiding me through what I should be doing to get this to work?

[Xanthir]

The problem you're having is that the login process is explicitly designed to not be accessible like this - it uses some CSRF (cross-site request forgery) protection to help guard against someone directly sending data at the form's action url without actually going thru the login page. If you check out the login page itself, you'll note there are some <input type=hidden> elements containing additional information.

This is actually meant to prevent something like a site making itself *look* like it has a form to do something else, but it's actually going to submit to the XKCD login page when the user types in their data (and then the site can do something nefarious once you're logged in). You can get around it from the server-side tho - you have to issue a GET request at the login page, then parse it as HTML and extract all the inputs, *then* use that data to actually make the POST request against the login url.

[ivnja]

Thanks - that makes sense, and I think I see the path forward.