automating youtube browsing

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

Garbeld
Posts: 6
Joined: Tue Sep 14, 2010 2:06 am UTC

automating youtube browsing

Postby Garbeld » Mon May 12, 2014 3:16 pm UTC

(Would this have been better in The Help Desk?)

I've got a list of bunches of Youtube channels, and am looking for a way to automatically open each video, record the URL, submission date, and author's comment.

At this point, I don't even know what languages are (most easily) relevant, though JavaScript (which I've not used before) seems a good guess. This is all well outside of my normal experience, as my past experiences trying to automate tedious processes have generally been less than simple and successful.
(... batch and make terrify me)

I've made some short hunts looking into scripting browser actions with Lua, with no obvious results - imagining something along the lines of navigating to page, copying URL+HTML source, and finding the tags enclosing the desired data... but even then, I'm pretty sure there's not going to be any <thisisthesubmissiondate></titsd> anywhere. In the past, I've failed as often as not in trying to find data in page sources by hand, let alone scripting data fetching, and Youtube certainly doesn't seem the most conducive to this kind of thing, what with all of its fancy features and such.

User avatar
Mat
Posts: 414
Joined: Fri Apr 21, 2006 8:19 pm UTC
Location: London

Re: automating youtube browsing

Postby Mat » Mon May 12, 2014 6:07 pm UTC

Have you looked at their API? You should be able to get all that information without needing to scrape the HTML.

They have client libraries for a bunch of languages but you could use Lua and parse the JSON yourself if that's easier.

Derek
Posts: 2176
Joined: Wed Aug 18, 2010 4:15 am UTC

Re: automating youtube browsing

Postby Derek » Mon May 12, 2014 7:19 pm UTC

I've used the Youtube Python API for something like this, and it's pretty straightforward.

Garbeld
Posts: 6
Joined: Tue Sep 14, 2010 2:06 am UTC

Re: automating youtube browsing

Postby Garbeld » Tue May 13, 2014 12:05 am UTC

Whoop whoop, thanks!

I've spent the however many hours since Derek's reply hunting through the Youtube Data API documentation and having no idea what I was doing, and as of an hour or so ago, finally figured out... well, kind of what I was doing. Enough to start getting somewhere at least.
That hour or so later, I now have the nine shiny, critical lines of code that allow me to work on the parts I do know: loops, trivial string manipulation, and writing text to a file!

Code: Select all

playlist_id = youtube.channels().list(part="contentDetails",forUsername=channel_name).execute().get("items")[0].get("contentDetails").get("relatedPlaylists").get("uploads")
playlist = [youtube.playlistItems().list(part="snippet",playlistId=playlist_id,maxResults=50).execute()]
nextToken = playlist[0].get("nextPageToken")
while nextToken:
  playlist.append(youtube.playlistItems().list(part="snippet",playlistId=playlist_id,maxResults=50,pageToken=nextToken).execute())
  nextToken = playlist[-1].get("nextPageToken")

Code: Select all

playlist[x].get("items")[y].get("id")
playlist[x].get("items")[y].get("snippet").get("title")
playlist[x].get("items")[y].get("snippet").get("description")

Derek
Posts: 2176
Joined: Wed Aug 18, 2010 4:15 am UTC

Re: automating youtube browsing

Postby Derek » Wed May 14, 2014 9:20 pm UTC

Yeah as I recall the documentation for the public API is shit, but once you figure it out it will give you pretty much all the information you need.

Garbeld
Posts: 6
Joined: Tue Sep 14, 2010 2:06 am UTC

Re: automating youtube browsing

Postby Garbeld » Thu May 15, 2014 12:06 am UTC

Eh, all of the specific functions calls I had to make seemed easy enough to understand, reading about. Didn't really have much more problem with that part than I do with most things.
The part that was making it difficult was figuring out what I had to download, what I had to do on their website, what included files, etc. I had to deal with.

That, and learning Python datatypes. After having gotten used to Lua's "tables, tables everywhere!", it's rather disorienting having to deal with dicts and lists and whatever others I didn't actually have to use for that program but had to read about to determine the irrelevancy of.


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 7 guests