Good books for learning to create proxy servers?

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

jacques01
Posts: 42
Joined: Thu Oct 08, 2015 4:56 am UTC

Good books for learning to create proxy servers?

Postby jacques01 » Tue Jun 21, 2016 2:32 am UTC

How can I learn to create proxy server code?

I can't seem to find any books dedicated to this specifically.

My purpose is the ability to hide or manipulate where a request is coming from, e.g. I want to make it look like I'm surfing the web in Wyoming when I'm actually in Florida.

I know it's possible to purchase such services, but I can also program and use Python and Java, and I'd like to learn some of the fundamentals in a do-it-yourself project.

Any recommended books? I can literally get any book, as I have free access to top universities libraries.

Tub
Posts: 296
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Good books for learning to create proxy servers?

Postby Tub » Tue Jun 21, 2016 8:41 am UTC

Proxy server software is available free and open source, there's no need to program anything yourself. However, you cannot just run a piece of software on your computer and it'll suddenly appear as if you were in wyoming. You actually need to own a computer in wyoming and route all your data through it. The services you can purchase do not charge you for the proxy server software, they charge you for their server hardware, bandwidth, maintenance and support. Programming your own software will not make it any cheaper.

If you want to learn more, I suggest starting with an introductory book on TCP/IP. If you just need a simple HTTP proxy, follow with a tutorial on the HTTP protocol. That should give you everything you need to understand how a proxy works, and how to program one yourself.
If you need a full VPN, that's a bit much for a do-it-yourself-project, but you can follow the TCP/IP book by studying the openvpn project.

Note that many geo-blocked sites (like netflix) will outright ban known proxy services, and proxies can neither prevent user tracking, nor can they give you the anonymity required for illegal activities. Lacking both privacy and anonymity, the only actual use I've found so far is occasionally unlocking geo-blocked youtube videos.

jacques01
Posts: 42
Joined: Thu Oct 08, 2015 4:56 am UTC

Re: Good books for learning to create proxy servers?

Postby jacques01 » Wed Jun 22, 2016 1:54 am UTC

However, you cannot just run a piece of software on your computer and it'll suddenly appear as if you were in wyoming. You actually need to own a computer in wyoming and route all your data through it.


Why is this the case? From a theoretical standpoint, all you need is a unique address in order to communicate between two entities, i.e. a unique bit string. Why does the physical location have to be exchanged in any handshakes / message communication?

proxies can neither prevent user tracking, nor can they give you the anonymity required for illegal activities. Lacking both privacy and anonymity, the only actual use I've found so far is occasionally unlocking geo-blocked youtube videos.


I must misunderstand how proxies work. Here is how I think they work, and then I would be grateful for explaining why my thinking is wrong.

Normally when I browse the web, I send / receive requests from my device A--I don't know how A is addressed--is it via IP, etc., but that should not matter. Let's call the server/website I interact with B, e.g. whatever is serving requests on www.wikipedia.org.

Without any proxy, the communication at a high level is A --> B. B knows everything about A, and the IP address, otherwise B couldn't get back to A.

Now, let's say A is located in Florida. A wants to appear as if he's located in Wyoming. A uses a proxy called C. C is located in Wyoming.

A forwards all his requests to C. C then forwards these to B.

From B's point of view, it appears that it's just C whose making connections and requests. It is impossible for B to know that C is actually tunneling back requests back and forth to A. The only way B could infer that A is acting through C as a proxy is by gathering statistics on use patterns (e.g. when 'C' uses B, when 'C' logs out, etc.). I exclude that case, as I'm not trying to solve that problem yet.

Now, how does this "proxy" scenario not provide anonymity and security? C doesn't have to tell B that's he's a proxy server. He appears just as any other regular user.

Tub
Posts: 296
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Good books for learning to create proxy servers?

Postby Tub » Wed Jun 22, 2016 8:25 am UTC

jacques01 wrote:Why does the physical location have to be exchanged in any handshakes / message communication?

There is no exchange of real-world locations in the TCP protocol. Read a TCP book. Read Wikipedia.

From B's point of view, it appears that it's just C whose making connections and requests. It is impossible for B to know that C is actually tunneling back requests back and forth to A.

Only if the proxy is correctly configured. Some will still leak information about A to B.
Also, the proxy provider knows your real-world information for billing purposes, so when the police or secret service asks the proxy provider about illegal activities from C, they're sure to turn over everything they have on A. So there goes your anonymity.

Privacy is not enhanced by proxy servers, as most forms of user tracking (cookies, browser fingerprinting etc) are not affected.

Security is not enhanced by proxy servers. You can still download malware via proxy, and your browser's security vulnerabilities are still there.

Yes, I'm aware that some proxy providers claim otherwise. That's just lies for marketing purposes. In fact, both privacy and security may be worse with a proxy.


The only thing worse than poor security is a false sense of security. Before you go playing around with proxies, learn the basics.

korona
Posts: 495
Joined: Sun Jul 04, 2010 8:40 pm UTC

Re: Good books for learning to create proxy servers?

Postby korona » Wed Jun 22, 2016 6:40 pm UTC

A proxy can certainly enhance privacy if you trust the proxy not to log your access. However a proxy alone is not enough to be truly anonymous.

jacques01
Posts: 42
Joined: Thu Oct 08, 2015 4:56 am UTC

Re: Good books for learning to create proxy servers?

Postby jacques01 » Thu Jun 23, 2016 12:12 am UTC

Only if the proxy is correctly configured. Some will still leak information about A to B.


Privacy is not enhanced by proxy servers, as most forms of user tracking (cookies, browser fingerprinting etc) are not affected.


a proxy alone is not enough to be truly anonymous.


What should I be researching then, that provides anonymity which apparently a proxy does not?

I'm more interested in being able to create new internet identities and maintain these without having to get a remote/virtual machine for each identity, in a way that is scalable and cost effective. The anonymity comes from the fact that while it may appear there are 100 unique identities, in fact it's just one guy (me) who's controlling them each. When an internet identity is no longer productive (e.g. it gets blacklisted), I need to be able to ideally create a new one programmatically and on the fly.

I am not an email spammer, but I will use that as an example for what I mean (of course, spammers probably buy the services of a company that sends the email for them, and don't have to go through these steps).

A spammer sends email from a server that has an IP address. At some point, the machine with that IP address will probably eventually get blocked by any SMTP servers, especially when it has too high of a volume, too many bounces, or the users mark the content as spam too much.

So, a spammer will need to able to get new IP addresses on the fly, that aren't associated in any way with their previous IP address.

Assuming IP is the only identifying factor, a spammer gets a new identity for their server each time they get a new IP.

I would be looking to do something similar, and it may be just as simple as changing IP addresses when needed (e.g.the current one is blacklisted). The fact that spammers are successful, and there are literally dozens of email campaign companies which are profitable tells me that this is very possible to do. Because the market is also saturated, it's probably not too difficult to do either.

What techniques or technology could provide me these functionalities? Is it enough then to just be able to swap the IP to get a new identity? I would even appreciate pointers to books or relevant articles.

User avatar
Xanthir
My HERO!!!
Posts: 5199
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Good books for learning to create proxy servers?

Postby Xanthir » Thu Jun 23, 2016 1:56 am UTC

This sounds incredibly sketch and I'm no longer comfortable even helping with it.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
phlip
Restorer of Worlds
Posts: 7542
Joined: Sat Sep 23, 2006 3:56 am UTC
Location: Australia
Contact:

Re: Good books for learning to create proxy servers?

Postby phlip » Thu Jun 23, 2016 5:00 am UTC

Yeah, you're being incredibly cagey with what you're actually trying to do...

We're not going to just sit here and help you try vainly not to get caught at whatever nonsense you're planning to get up to.

I highly recommend either explaining what you're trying to do, if it's legitimate, or just leaving, if it's not.

Code: Select all

enum ಠ_ಠ {°□°╰=1, °Д°╰, ಠ益ಠ╰};
void ┻━┻︵​╰(ಠ_ಠ ⚠) {exit((int)⚠);}
[he/him/his]

jacques01
Posts: 42
Joined: Thu Oct 08, 2015 4:56 am UTC

Re: Good books for learning to create proxy servers?

Postby jacques01 » Thu Jun 23, 2016 11:53 pm UTC

Yeah, you're being incredibly cagey with what you're actually trying to do...


I'm trying to avoid specifics in order to get a general (theoretical) explanation of what technology or techniques I'm referring to. That way I can begin researching on my own.

We're not going to just sit here and help you try vainly not to get caught at whatever nonsense you're planning to get up to.


I am not asking for working code examples. I am only looking for pointers to books, resources, etc. And relevant keywords to whatever technology or techniques I'm referring to so I know what field of study this is.

I highly recommend either explaining what you're trying to do, if it's legitimate, or just leaving, if it's not.


I'm interested in being able to do a few things...

1. Be able to verify a given email. That requires an SMTP handshake. Too many of these can cause the server to blacklist (possibly permanently) the offending IP.

2. Develop reliable APIs around existing services that don't publicly offer an API, or if they do, it's not free. For example, Google search will begin to send captchas if it thinks the user is a robot or sending too many search requests.

I am a hobbyist and not running a large evil corporation.

I'm looking for discussion with fellow engineers and programmers.

User avatar
jestingrabbit
Factoids are just Datas that haven't grown up yet
Posts: 5959
Joined: Tue Nov 28, 2006 9:50 pm UTC
Location: Sydney

Re: Good books for learning to create proxy servers?

Postby jestingrabbit » Fri Jun 24, 2016 5:38 pm UTC

I think, for the first one, you want to be transparent with your user. Which means sending an email that they then demonstrate receiving by following some particular link, or inputting some particular data. Trying to sidestep this isn't sensible imo. Sure, formatting and sending emails that work with a broad section of the existing services sucks, but you can go plain text till it makes sense to upgrade to fancy formatting.

The second thing you want to do is basically bilk people who are providing a free service (which they pay for by getting people to pay for more heavy duty usage) into providing you with something they want you to pay money for. This is clearly dodgy, by any reasonable assessment.

So, apart from the suggestion to look into the details of the TCP/IP layers, I doubt you're going to be given much help here.
ameretrifle wrote:Magic space feudalism is therefore a viable idea.

User avatar
Flumble
Yes Man
Posts: 1923
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Good books for learning to create proxy servers?

Postby Flumble » Fri Jun 24, 2016 6:53 pm UTC

jacques01 wrote:I'm trying to avoid specifics in order to get a general (theoretical) explanation of what technology or techniques I'm referring to. That way I can begin researching on my own.

How about you tell us the specifics so we can reply with general explantions and pointers (as you've made it clear you don't want a specific solution)?

jacques01 wrote:2. Develop reliable APIs around existing services that don't publicly offer an API, or if they do, it's not free. For example, Google search will begin to send captchas if it thinks the user is a robot or sending too many search requests.

Apart from what jestingrabbit says above, the country you live in may have laws regarding this. For example, in the Netherlands it's allowed to scrape databases (for as far as the database manager allows you to) and even to make these available to others as long as the database is not a key selling point of that copmany/organization. The best thing you can to is find out what the laws are in your country and abandon this topic if it's not deemed legal. The worst thing you can do is continue regardless and keeping your questions vague and sketchy. :wink:

Speaking of vague, maybe it's just me, but I have no idea what your first question means. Proper email verification is built into modern mail servers, AFAIK, and has nothing to do with using a single IP.

Breakfast
Posts: 117
Joined: Tue Jun 16, 2009 7:34 pm UTC
Location: Coming to a table near you

Re: Good books for learning to create proxy servers?

Postby Breakfast » Fri Jun 24, 2016 8:04 pm UTC

Flumble wrote:
jacques01 wrote:2. Develop reliable APIs around existing services that don't publicly offer an API, or if they do, it's not free. For example, Google search will begin to send captchas if it thinks the user is a robot or sending too many search requests.


...

For example, in the Netherlands it's allowed to scrape databases (for as far as the database manager allows you to) and even to make these available to others as long as the database is not a key selling point of that copmany/organization.


And to expand on this, the, "as long as the database is not a key selling point...," part is, at best, fuzzy and is the kind of thing that a company will argue about in court when they sue you.

So, like others have said, you can keep trying to do whatever you're trying to do but you probably won't get any help here and there very well could be legal ramifications that you're not ready for even if it's just toeing the line of possibly, maybe not legit.

User avatar
jestingrabbit
Factoids are just Datas that haven't grown up yet
Posts: 5959
Joined: Tue Nov 28, 2006 9:50 pm UTC
Location: Sydney

Re: Good books for learning to create proxy servers?

Postby jestingrabbit » Sat Jun 25, 2016 9:03 am UTC

Flumble wrote:Speaking of vague, maybe it's just me, but I have no idea what your first question means. Proper email verification is built into modern mail servers, AFAIK, and has nothing to do with using a single IP.


The way I see it, there are a few different questions that a site could be intending to answer when it sets out to verify an email address. In order of complexity, these are

1. Does the email match (some reasonable facsimile of) the email standard?
2. Does a mailbox with that address actually exist on some mail server?
3. Does the user own, and have access to, that mailbox?

To answer these you need (1) a (fiendishly complicated to get it right) regex, (2) to communicate with a mail server, and (3) to get your user to do some of the work. irl, you actually want to answer question 3.
ameretrifle wrote:Magic space feudalism is therefore a viable idea.

elasto
Posts: 3065
Joined: Mon May 10, 2010 1:53 am UTC

Re: Good books for learning to create proxy servers?

Postby elasto » Sat Jun 25, 2016 4:59 pm UTC

Also, assuming you aren't doing anything high bandwidth, and taking your word for it that it's legitimate, couldn't you just use a public service like TOR?

Wikipedia wrote:Tor is free software for enabling anonymous communication. The name is derived from an acronym for the original software project name "The Onion Router".[7][8] Tor directs Internet traffic through a free, worldwide, volunteer network consisting of more than seven thousand relays[9] to conceal a user's location and usage from anyone conducting network surveillance or traffic analysis. Using Tor makes it more difficult for Internet activity to be traced back to the user: this includes "visits to Web sites, online posts, instant messages, and other communication forms".[10] Tor's use is intended to protect the personal privacy of users, as well as their freedom and ability to conduct confidential communication by keeping their Internet activities from being monitored.

Onion routing is implemented by encryption in the application layer of a communication protocol stack, nested like the layers of an onion. Tor encrypts the data, including the destination IP address, multiple times and sends it through a virtual circuit comprising successive, randomly selected Tor relays. Each relay decrypts a layer of encryption to reveal only the next relay in the circuit in order to pass the remaining encrypted data on to it. The final relay decrypts the innermost layer of encryption and sends the original data to its destination without revealing, or even knowing, the source IP address. Because the routing of the communication is partly concealed at every hop in the Tor circuit, this method eliminates any single point at which the communicating peers can be determined through network surveillance that relies upon knowing its source and destination.


That's if a VPN service you can pay a few dollars a month for isn't any good. For me that works just fine.

Tub
Posts: 296
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Good books for learning to create proxy servers?

Postby Tub » Sun Jun 26, 2016 11:31 am UTC

jacques01 wrote:2. Develop reliable APIs around existing services that don't publicly offer an API, or if they do, it's not free. For example, Google search will begin to send captchas if it thinks the user is a robot or sending too many search requests.

Legally getting access to a sufficient amount of public IPs to make your access pattern seem unrelated will cost you more than just buying an API key in the first place. And your money will go to some unrelated proxy services instead of the service you actually rely on. Please play nice with the web services you are using. And implement sensible result caches.

elasto wrote:Also, assuming you aren't doing anything high bandwidth, and taking your word for it that it's legitimate, couldn't you just use a public service like TOR?

That is dangerous advice. Unless you have a very good understanding of what TOR does and doesn't do for you, stay as far away from it as you can. You simply cannot trust the exit nodes.

elasto
Posts: 3065
Joined: Mon May 10, 2010 1:53 am UTC

Re: Good books for learning to create proxy servers?

Postby elasto » Sun Jun 26, 2016 12:37 pm UTC

Tub wrote:That is dangerous advice. Unless you have a very good understanding of what TOR does and doesn't do for you, stay as far away from it as you can. You simply cannot trust the exit nodes.

But trusting the exit node didn't seem to be of particular consideration for the OP; The priority was solely the deception of the final destination.

korona
Posts: 495
Joined: Sun Jul 04, 2010 8:40 pm UTC

Re: Good books for learning to create proxy servers?

Postby korona » Mon Jun 27, 2016 5:57 pm UTC

I'm never used Tor but I don't see why you have to trust exit nodes. My understanding is that no one is able to trace your ip as long as there is at least one node in the path between you and the server that can be trusted. Of course exit nodes can spy on your unencrypted traffic but which security-critical websites do not use HTTPS anyways?

User avatar
hotaru
Posts: 1019
Joined: Fri Apr 13, 2007 6:54 pm UTC

Re: Good books for learning to create proxy servers?

Postby hotaru » Mon Jun 27, 2016 6:08 pm UTC

korona wrote:Of course exit nodes can spy on your unencrypted traffic but which security-critical websites do not use HTTPS anyways?

exit nodes can also do DNS poisoning and do SSL stripping attacks, as well as a number of attacks against SSL/TLS itself in the case of an out of date or poorly configured server (which are unfortunately quite common).

Code: Select all

factorial product enumFromTo 1
isPrime n 
factorial (1) `mod== 1

Tub
Posts: 296
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Good books for learning to create proxy servers?

Postby Tub » Mon Jun 27, 2016 7:24 pm UTC

korona wrote:as long as there is at least one node in the path between you and the server that can be trusted.

I'll just remark that that is an interesting assumption you're making and leave it at that. There have been papers about protocol vulnerabilities too, but I'm too lazy to look them up right now.

korona wrote:Of course exit nodes can spy on your unencrypted traffic

Yes, that's one of the things one has to understand before using TOR. You say "of course", I say: most people, likely including the OP, are not aware, and should thus stay away from TOR.
By using TOR, you're voluntarily letting yourself get man-in-the-middle'd by an unknown and probably malicious entity. It takes a lot of knowledge, preparation and care to go up against that, even with https.

My advice stands: stay away from TOR unless you have a very good understanding of the workings and drawbacks.

Besides, I would imagine that a system that funnels many users through few exit nodes would cause those exit nodes to hit rate limits on the popular APIs, too, so there's nothing gained for the OP.


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 3 guests