The Red Spider Project

[Zarq]

That last one sounds like what I was thinking about. Proven once again that I have no original ideas.

[Jplus]

Actually I think it's not exactly the same, and with some discussion the ideas should be able to exist alongside.

Your idea: a meta-text adventure that lets you navigate between text adventures as if they're houses (if I understood your idea correctly).

FireRogue's idea: a variant of rsshell that keeps track of what commands you're running and your results in the various rsp games. Based on those things, it fires up achievement messages. Rather an RPG than a text adventure.

[Jplus]

A quick update of what's going on.

• Upcoming new commands: summon and level_up.
• FireRogue and I had an interesting discussion on the teleportation behaviour of rsshell.
• summon and rsshell need testing (see links in the previous bullets). Help is welcome.
• FireRogue and scared are working on an overhaul of xkcd-fetch.
• Xenomortis is still working on his two-coloured topological game of life.
• We could use help from a CMake wizard for issues #24 and #33.
• zed0's early advshell and advbrowser are free to be picked up by anyone for further development.
• Nowadays the IRC channel is almost permanently populated, mostly thanks to zed0. Depending on chance and the time of the day, you're also likely to find me, mrhmouse (who fixed our shebang lines in August) and FireRogue.
• Latest additions to the wiki: minimal requirements and project directory overview.
• We're still awaiting the first proposal for a six-legged ASCII art spider.
• Latest addition to the list of free ideas: Petrovich (with permission of the inventor).

Edit:

• cjdrum's suggestions have also been added to the list of free ideas.
• I rewrote the Readme and companions. Please take a look and give your opinion.

[Xenomortis]

level up should include Assembler.
Just sayin'.

[ahammel]

Thanks to Julian for doing the final clean up for level_up!

Re: assembly language. I'm open to suggestions, but there are a lot of assembly languages, no universal conventions on filenames as far as I can tell (and if you think I'm doing any kind of inspection of the source code to figure out what language it's in, you're very much mistaken ), and I have no data on how many lines of code it takes to perform a given task in assembler compared to C.

[Dason]

and I have no data on how many lines of code it takes to perform a given task in assembler compared to C.

I suppose one could take a sample of a few C files and compile them down to assembly and compare the relative sizes to get an idea of that.

[Jplus]

Thanks to Julian for doing the final clean up for level_up!

You're welcome! Are you now going to pull request/merge it into the-xkcd-community:master? Or should I do that?

and I have no data on how many lines of code it takes to perform a given task in assembler compared to C.

I suppose one could take a sample of a few C files and compile them down to assembly and compare the relative sizes to get an idea of that.

The disadvantage of that is that library routines called from C will be part of the compiled assembly, while assmbly programmers would normally also just #include the relevant headers and then do the equivalent to a library call in C. So doing what Dason suggest here would over-estimate the number of lines for an assembly program, probably by a large margin.

A slightly more balanced alternative could be to compare solutions in C and assembly on the Project Euler forums. Prerequisite is that you solved at least a few problems yourself, otherwise you won't have access. I could "harvest" some solutions if you want, I solved about 50 problems.

@everyone: please have a look at my latest pull request, I really need some opinions there!
(It doesn't have to be a strong opinion, just an "I'm fine with this" would already be great.)

[alessandro95]

I solved 61 and I program in assembly myself so I can give you some data as well, thought rosettacode is probably much better than projecteuler to compare different languages

[FireRogue]

maybe lines of code isnt the right way to do this? esp because you'd need to write a basic tokenizer, especially for languages that support semicolons (quick problems in C: for loop declarations are 1 line or 3 lines, not 2 as would be counted by semicolons, multiple statement / line or statements split over lines, multiple assignment / statement chaining, using commas, etc) maybe we should be looking at this differently? count functions/methods/loops/classes/files written or something like that?

sorry that im bringing this up so late into the project...

[ahammel]

Thanks to Julian for doing the final clean up for level_up!

You're welcome! Are you now going to pull request/merge it into the-xkcd-community:master? Or should I do that?

Knew I was forgetting something. It's done.

The documentation rewrite looks good to me!

I solved 61 and I program in assembly myself so I can give you some data as well, thought rosettacode is probably much better than projecteuler to compare different languages

Rosetta code would be an ideal data source. I had plans to that effect once upon a time.

maybe lines of code isnt the right way to do this?

Oh, it's absolutely not the right way to do this, and I'm not even going so far as to count semi-colons. It is, however, the easiest way to do this, and since my intention is to provide a totally non-serious SLOC count that doesn't get wildly out of joint amongst different languages, I'm happy with it for now.

[Jplus]

Great! Our inventory of default-installed commands has grown for the first time in a very long while.

Also, thanks for the feedback on the documentation rewrite!

A suggestion for level_up: you could count the gzipped file size of the source code files (they do that in the benchmarks game). With Python goodness that might be only slightly harder than counting the number of lines of code. On the other hand, I think counting lines of code is fine too since people generally agree on how much code should be on a single line in a given language. Only code golfers and obfuscators tend to go against those conventions.

[ahammel]

A suggestion for level_up: you could count the gzipped file size of the source code files (they do that in the benchmarks game). With Python goodness that might be only slightly harder than counting the number of lines of code.

Certainly another option. I think I was worried about performance when I started the project, but some benchmarking I did just now suggests that wouldn't be the problem. Gzipping 5000 lines of python in 70 files to /dev/null is negligible in terms of time cost.

On the other hand, I think counting lines of code is fine too since people generally agree on how much code should be on a single line in a given language. Only code golfers and obfuscators tend to go against those conventions.

And I have no problem discouraging people from trying to cram too much on to a line.

[Jplus]

In the past two days, two big merges went into the integration branch. The first added Alex' level_up command, the second made big improvements to the Readme and the other documentation files. I recommend that all hackers pull in the new changes. Here's a quick how-to:

(I assume that your "upstream" remote points to git://github.com/the-xkcd-community/the ... roject.git ; otherwise substitute the correct remote name.)

Code: Select all

git checkout master
git pull upstream master
# if you have more branches, for each do the following:
git checkout <other branch>
git merge master

With the pull and with each merge you might walk into merge conflicts. Here are two different guides that explain how to handle a merge conflict. It's not as scary as it sounds!

Note that you need Python 3 and Tcl/Tk for the level_up command.

[ahammel]

I was looking at level_up a bit this weekend and I've managed to remove the py3k requirement with a little bit of import magic. It should work perfectly fine with any Python>=2.7 now.

[Lleu]

Talked to Julien forever ago about contributing to the RSP, and just sent my first ever pull request! I feel so professional....

Anyway, just saying hi. I might jump on one of the free ideas. Or see if I can figure out how to fix one of the issues. Or something else entirely. I don't know. I'm just here to have fun and learn Python!

[Jplus]

Yay, well done Lleu!

There's only one more thing that needs to be done before we can pull your program. The filename has to be added to the executable_scripts variable in setup.py, on line 32. That will cause it to be installed when people run the setup script. You can do this yourself. You could also let me do it, but it will certainly take more than a week before I find the time and resources.

Somewhat less pressing is the fact that if we leave the "What" section in the Readme unchanged, it will seem to suggest that your command doesn't exist. That can be solved either by adding your command to the list or by changing the wording of that section so it doesn't imply that the programs listed there are the only ones. Either way, ideally the Readme should be updated and again that's something you could do by yourself.

Finally (the least important), the "state of the art" page on the wiki should also be updated. You can do this yourself as well, but there's no haste because the date of last update is listed at the top of the page.

[FireRogue]

as the list of all the little bits and pieces we need to update to introduce a command start piling up, it might be worth adding an unlisted dev command we can run to do it all for us

[Jplus]

as the list of all the little bits and pieces we need to update to introduce a command start piling up, it might be worth adding an unlisted dev command we can run to do it all for us

That's a great idea. The summon command might come in handy there.

[Lleu]

Fixed ./setup.py and the Readme. I'll take care of State of the Art after work.

Anyway, I'm currently reading through Git's manuals and documentation to figure out how best to handle Issue # 35: Git home-made convenience functions.

[Jplus]

@Lleu: It seems that with two little fixes, your program will be ready for merging. Good job and sorry for the huge delay!

[whaatt]

Hi everyone, I haven't lurked around these forums in a while and only now saw this project, which I think is really cool.

Sorry if this is OT, but here's my idea, for whatever it's worth: a command that is reminiscent of the fortune command on some old Unix boxes, except that all of the fortunes are snippets of (witty?) posts from these forums. If that's not quite feasible to do, maybe just hand-made fortunes with xkcd references. What do you all think?

Feel free to jump on this idea if you like it; otherwise I'll fork this repo and give it a shot in the next few days or so.

[FireRogue]

You'd have to do some really fancy coding or rely on an xml library as a dependency, but it should be possible to get a random forum page, scrape for text boxes and grab things from them. how you classify 'witty' is up to you, but using quotes in signatures might be a good place to start.

[Jplus]

Hi everyone, I haven't lurked around these forums in a while and only now saw this project, which I think is really cool.

Sorry if this is OT, but here's my idea, for whatever it's worth: a command that is reminiscent of the fortune command on some old Unix boxes, except that all of the fortunes are snippets of (witty?) posts from these forums. If that's not quite feasible to do, maybe just hand-made fortunes with xkcd references. What do you all think?

Feel free to jump on this idea if you like it; otherwise I'll fork this repo and give it a shot in the next few days or so.

Welcome whaatt! Your idea is really nice, we (you) should definitely do that.

[Lleu]

Is it better practice to wait until something substantial is complete before submitting a pull request, or to do a little bit at a time?

[FireRogue]

commit to your own branch in small bits, but the only thing that should be submitted as a pull request is completed and functional code that has been bug tested. (Don't worry if there are still some bugs, we'll iron them out later.) It doesn't matter how much code you submit so long as it works

[Jplus]

Preferably submit a pull request for every tiny bit that you complete, because otherwise you might accumulate many bits of nearly finished stuff that never get submitted. Mind you, it won't be actually pulled until it actually works.

[FireRogue]

I defer to Jplus because his response is better

[Dason]

You'd have to do some really fancy coding or rely on an xml library as a dependency, but it should be possible to get a random forum page, scrape for text boxes and grab things from them. how you classify 'witty' is up to you, but using quotes in signatures might be a good place to start.

Might just be better to manually compile a list of quotes. Not nearly as flashy but I'm guessing this is how the fortune package came to be. I actually wrote an R package that provides similar functionality (within R) grabbing quotes from a different forum I'm a part of. Once I got other people to recommend quotes the database of quotes was a decent size.

[whaatt]

You'd have to do some really fancy coding or rely on an xml library as a dependency, but it should be possible to get a random forum page, scrape for text boxes and grab things from them. how you classify 'witty' is up to you, but using quotes in signatures might be a good place to start.

Might just be better to manually compile a list of quotes. Not nearly as flashy but I'm guessing this is how the fortune package came to be. I actually wrote an R package that provides similar functionality (within R) grabbing quotes from a different forum I'm a part of. Once I got other people to recommend quotes the database of quotes was a decent size.

So I've started working on the command, and I found that the XKCDB site is a treasure trove of clever exchanges from the #xkcd IRC channel. Thoughts on this? They aren't one-line aphorisms in the sense of the original fortune command, but some solid content nonetheless.

I thought about two possibilities: The first was that I could write a scraper to get a bunch of these into a JSON or text file that the command would randomly pick from, and this would be a lot like the original command. The other option, which I'm currently leaning towards, is dynamically accessing the "random" URL on the XKCDB site, and scraping the first exchange on the page. This, of course, would require internet access, but I think it would be future-proof in the sense that we wouldn't have to update a file whenever we wanted to add more IRC exchanges.

A slightly related question: what's the current consensus on using non-library modules (e.g. BeautifulSoup for HTML scraping) in Red Spider programs? Are these things that would be left to the user to install (i.e. listed under dependencies), or are these things that could be included with program code itself (i.e. in the same folder)?

[FireRogue]

maybe we could make a /dep folder for dependencies assuming they're not installed on the user's computer?

also, what about a hybrid approach? preloaded file, and if there's internet the program randomly decides whether or not to scrape, cache, and display a new quote? if there's no internet or it just decides to do it, it goes directly to the file.

[Jplus]

As far as the caching of quotes is concerned, I think the best way to go is to scrape the website once to create an offline cache for when the user is disconnected from the internet, but use the online random url for the actual quote to display whenever the user is connected. You can give the user the option to occasionally update the offline cache, or do it automatically periodically.

As for BeautifulSoup, since it's got quite a few dependencies of its own, including a copy in our source tree isn't really an option. For the time being I think it's fine to request that users install it by themselves, especially because BeautifulSoup may also be used for xkcd-fetch and possibly other commands. In the future we can make external dependencies more convenient by having the setup script automatically install them.

So, more or less what FireRogue said, but with a few differences.

[Xanthir]

Don't rely on BeautifulSoup - it's not a good HTML parser these days. Use html5lib (installable via pip) - it's a direct implementation of the HTML spec's parsing algorithm. You can then use any of several tree libraries - BeautifulSoup's tree is an option, but I use lxml because it has a nice library that goes on top, cssselect, which lets you run CSS selectors over the document rather than XPath or just manual tree-walking.

[whaatt]

As far as the caching of quotes is concerned, I think the best way to go is to scrape the website once to create an offline cache for when the user is disconnected from the internet, but use the online random url for the actual quote to display whenever the user is connected. You can give the user the option to occasionally update the offline cache, or do it automatically periodically.

Sounds good. Where should this sort of cache file be stored in the directory structure?

[Jplus]

Assuming your command is called "fortune": work/fortune/.

Xanthir: thanks for pointing that out.

[whaatt]

So I finally got around to coding this up today, and I've uploaded what I've done at my GitHub fork. I ended up going with LXML for the parsing, particularly since, as Xanthir helpfully noted, LXML supports the cssselect module for using CSS selectors.

As for the caching mechanism: There is a cache sub-command/parameter that is accepted to generate the cache file from one or more pages of the top threads on XKCDB. Whenever a random or specific (by ID) fortune is requested, the program tries to query the online source and falls back to the cache file if necessary, then appends the fortune to the cache file if it doesn't already exist in there. I devised a way to do this such that the large cache file does not need to be read fully into memory for these functions to work.

However, I'm still confused as to where the cache file should reside. Is the work dir in relation to the root directory of Red Spider, or is it in relation to /src/? In other words, what should I pass as the argument to open() in Python? Right now I have the fortune file in the same directory as my code.

I've included a help string in my source for more information on accepted parameters. Thanks!

[Jplus]

However, I'm still confused as to where the cache file should reside. Is the work dir in relation to the root directory of Red Spider, or is it in relation to /src/? In other words, what should I pass as the argument to open() in Python? Right now I have the fortune file in the same directory as my code.

It depends a bit on the purpose of the example file. If it's just an example (i.e. documentation) for fellow coders so they know how to generate a fortune file, then it should be in $RED_SPIDER_ROOT/doc. If it's actually meant to be used by your program until the user runs "fortune cache" it should be stored in $RED_SPIDER_ROOT/other (because it's a non-program, non-documentation file that still needs versioning) and then copied to $RED_SPIDER_ROOT/work/fortune upon installation. Either way, the cache file that your program is actually going to use should be stored in the latter location. Please ask for help if you don't know how to find the project root from within your program!

So yes, work relative to the project root. If that directory doesn't exist in your copy you probably didn't run setup.py.

[whaatt]

So yes, work relative to the project root. If that directory doesn't exist in your copy you probably didn't run setup.py

D'oh! That's it. And the xkcd-fetch.py source code was pretty useful to figure out the Red Spider root directory stuff in Python.

Thanks everyone for the help; I've submitted a pull request for this command, but I probably (read: certainly) goofed up somewhere along the way, so please let me know what needs to be changed or if you see a more efficient way to do something, and I'll incorporate the suggestions.

[whaatt]

So yes, work relative to the project root. If that directory doesn't exist in your copy you probably didn't run setup.py.

Actually, for some reason, setup.py creates all of the other directories, like /extbin and /bin, but /work doesn't show up. Is there something more to be done, or are individual programs supposed to create the directory if it doesn't already exist?

[FireRogue]

iirc, isnt work supposed to be a temporary directory for the install that gets deleted at the end? mind you, i dont have the actual code in front of me. if that's not the case, it sounds like a bug

[Jplus]

Work is never deleted because it's the user's working directory in which they can store any files they want. It might not be created by your (whaatt's) version of setup.py; I might have introduced that feature in a branch that hasn't been merged yet. In that case you can just create the directory if it doesn't exist yet, as you said. We'll remove that redundancy when the new version of setup.py is merged.