Bitcoin-payable HTTP Proxy

Posted by Corentin Debains

Build a Bitcoin-payable HTTP Proxy

Overview

Suppose you are in an area (or a country) behind a firewall, and you want to access a web page outside that firewall. In theory, a proxy would be a simple solution: just route your requests through the proxy server to make the website think a user is located wherever the proxy is located. However, in practice, finding a good proxy with useful options and good performance can be tricky and often requires a monthly subscription for access. This requires creating an account and adding funds to it, usually with a credit card - and this significantly raises the fixed costs of using a proxy.

One solution is to set up 21 as a bitcoin-payable HTTP proxy, with (say) 1 cent in bitcoin charged per request. This would both compensate the proxy provider and reduce the up-front cost for the proxy user, as no account setup would be needed. The provider doesn't manage accounts and the user doesn't have a password to remember. Moreover, as a portable device with a virtualizable IP, requests from 21 are less likely to be blocked - which is potentially quite useful for a low-volume (<10000 requests per day) proxy server used by a few of your friends abroad. Let's see how to set one up.

Prerequisites

Install 21

You will need the following first:

If you've got all the prerequisites, you are ready to go. Let's get started!

Step 1: Install dependencies

You will need to install the requests library:

sudo pip3 install requests

Step 2: Set up the example server for debugging

First, we will set up a simple example server running locally. This will be used to test out the proxy that you are about to write.

To set up the example server open a terminal. If you use a 21 Bitcoin Computer, ssh into it:

## Only necessary if you use a 21 Bitcoin Computer
ssh twenty@IP_ADDRESS

Create a file called example-server-for-http-proxy.py

nano example-server-for-http-proxy.py

Fill it with the following code (also downloadable here):

#!/usr/bin/env python3
from flask import Flask, request

app = Flask(__name__)


@app.route("/")
def hello():
    return "Hello, World!\n"


@app.route("/AmIBehindProxy")
def areyouaproxy():
    if "X-Forwarded-For" in request.headers:
        return "Yes, You are behind a proxy\n"
    else:
        return "I didn't see any evidence of a proxy\n"

if __name__ == "__main__":
    app.run(host="::", port=5000)

This example-server-for-http-proxy.py has two endpoints (/ and /AmIBehindProxy). These endpoints will be used to debug your proxy locally and confirm that it's working as advertised before trying to connect to a real remote server like stanford.edu. Let's start the server to test it out:

python3 example-server-for-http-proxy.py

Leave the current terminal open and open up another one. If you use a 21 Bitcoin Computer, SSH into it:

## Only necessary if you use a 21 Bitcoin Computer
ssh twenty@IP_ADDRESS

Use the command line curl utility to request the URL http://[::1]:5000/ from this example server as follows:

curl -g -6 http://[::1]:5000/

It should display Hello, World!. Now, try curling the other endpoint of this example server:

curl -g -6 http://[::1]:5000/AmIBehindProxy

It should display I didn't see any evidence of a proxy. All the example server is doing is looking for the standard X-Forwarded-For header in the HTTP request to see if it was indeed a proxy that routed your request. We will use this to debug the proxy server below.

Step 3: Set up the HTTP Proxy

Leave the current terminal open and open up a third one. If you use a 21 Bitcoin Computer, SSH into it:

## Only necessary if you use a 21 Bitcoin Computer
ssh twenty@<IP Address>

Create a file called proxy.py

nano proxy.py

Fill it with the following code (also downloadable here):

#!/usr/bin/env python3
import requests

from flask import Flask, request, Response
from werkzeug.datastructures import Headers

from two1.wallet import Wallet
from two1.bitserv.flask import Payment

app = Flask(__name__)

wallet = Wallet()
payment = Payment(app, wallet)


@app.route('/', defaults={'path': ''})
@app.route('/<path:path>')
@payment.required(3000)
def catch_all(path):
    try:
        h = Headers(request.headers)
        h.add('X-Forwarded-For', request.remote_addr)
        h.remove('HTTP_BITCOIN_MICROPAYMENT_SERVER')
        h.remove('HTTP_RETURN_WALLET_ADDRESS')
        h.remove('Bitcoin-Transfer')
        h.remove('Authorization')
        h.remove('Content-Length')

        r = requests.request(
            method=request.method,
            url=request.url,
            headers=h,
            files=request.files if request.files else None,
            data=request.data if request.data else None,
            params=request.args if request.args else None,
            timeout=5
        )
    except (
        requests.exceptions.Timeout,
        requests.exceptions.ConnectTimeout,
        requests.exceptions.ReadTimeout
    ):
        return Response(status=504)
    except (
        requests.exceptions.ConnectionError,
        requests.exceptions.HTTPError,
        requests.exceptions.TooManyRedirects
    ):
        return Response(status=502)
    except (
        requests.exceptions.RequestException,
        Exception
    ) as e:
        if app.debug:
            raise e
        return Response(status=500)

    headers = list(r.headers.items())
    return Response(
        r.text if r.text else None,
        status=r.status_code,
        headers=headers
    )

if __name__ == '__main__':
    app.run(host='::', port=9000)

Let's start the proxy:

python3 proxy.py

The proxy is now running on port 9000 of your computer, and the destination server from the earlier step is running on port 5000.

Note: our proxy looks a lot like our original server, and that's normal! They are both web (HTTP) servers based on the Flask web microframework. But the proxy is a little more elaborate.

First, the proxy uses bitcoin. The Payment class is a blueprint designed to makes it very simple to use micropayments with our server endpoints. It is applied here as a decorator: @payment.required(3000), which means that every call to this endpoint will require a payment of 3000 satoshis. Your 21 wallet is used as the payout address for all the money collected by the proxy.

Second, the @app.route part is slightly different than the corresponding Python decorator on the example server. It catches requests made to the destination server and routes them to the destination server if a micropayment has been embedded in the request. The catch_all function performs the following operations:

  1. Insert the X-Forwarded-For header, which is a conventional proxy header.
  2. Remove the bitcoin micropayment headers to receive the payment.
  3. Call the destination website with the same parameters that the user gave, with some error handling to return common Proxy Error codes.
  4. Send the Response back with the destination server's reply.

Now we're going to put the pieces together.

Step 4: Create a client that pays bitcoin to a proxy to connect to a remote server

To set up the client, open a fourth terminal. If you have a 21 Bitcoin Computer, SSH into it:

## Only necessary if you have a 21 Bitcoin Computer
ssh twenty@<IP Address>

Create a file called client-for-http-proxy.py in a new directory.

nano client-for-http-proxy.py

Fill it with the following code (also downloadable here):

#!/usr/bin/env python3
from two1.wallet import Wallet
from two1.bitrequests import BitTransferRequests

EXAMPLE_SERVER = "http://[::1]:5000"
PROXY = "http://[::1]:9000"

proxies = {"http": PROXY}

wallet = Wallet()
requests = BitTransferRequests(wallet)


def main():
    print("Call the example server directly")
    print("The goal here is to confirm that the example server is \
    reachable and can distinguish between a proxied and non-proxied \
    connection.")
    r = requests.get(url=EXAMPLE_SERVER + "/AmIBehindProxy")
    print(r.text)

    print("Call the example debug server through the proxy, paying 3000 satoshis per request")
    print("The goal here is to confirm that the example server was hit through a proxy.")
    r = requests.get(url=EXAMPLE_SERVER + "/AmIBehindProxy", proxies=proxies)
    print(r.text)

    print("Now call a real server at stanford.edu by paying the proxy some bitcoin")
    r = requests.get(url="http://httpbin.org/get", proxies=proxies)
    print(r.text)

if __name__ == '__main__':
    main()

Finally, let's run the client:

python3 client-for-http-proxy.py

To recap, what we did here was:

  • set up an example server locally
  • set up a proxy
  • used the example server to test out whether the proxy worked properly
  • then made a request to a real remote server (stanford.edu) through the proxy

The bitcoin portion of this is based on the 21 Bitcoin Library's BitTransferRequests class, essentially a bitcoin-powered wrapper around the standard Python3 requests library. Thus, when the proxy returns a 402 Payment Required error, the library automatically handles the micropayment and passes through the full HTTP response object unscathed. In the sample code above we only used r.text, but you could try r.headers and r.status_code. They would be the same between the proxied and direct connection.

After you've done this, you can run 21 log to see the various changes to your buffer.

Next Steps

This proxy is fairly limited in what in can do and highly insecure! However, it's a great experiment to explore the power of bitcoin micropayments with existing tools and protocols. Here are some ideas on what to build next:

  • Make this example high performance by replacing the simple Flask proxy server with a proper deployment on the underlying hardware. See this and this for some first steps.

  • Many websites that are behind services like Cloudflare will reject requests from proxies. For educational purposes, see if you can increase the number of websites you can connect to via the proxy by removing the "X-Forwarded-For" headers, changing the "User-Agent" strings, and doing similar things to the outbound request.

  • Modify this code to create a paying proxy which would pay for bitcoin-payable resources on your behalf when you request them.

  • Modify this code to create a reverse proxy, which you can put in front of any existing HTTP-accessible resource to create a bitcoin paywall.

  • Integrate this example with a Chrome-based bitcoin wallet to create a simple example of what a machine-payable internet might look like.

  • Modify this example to route requests through a production proxy service like Hola.

If you build anything like this and want to earn some bitcoin for your efforts, write it up and submit it as a bitcoin tutorial. If we decide to publish it on our site, you'll win $200 in BTC! You can also come to our Slack channel at slack.21.co to find other 21 users to give you feedback. We look forward to seeing you there!


How to send your Bitcoin to the Blockchain

Just as a reminder, you can send bitcoin mined or earned in your 21.co balance to the blockchain at any time by running 21 flush . A transaction will be created within 10 minutes, and you can view the transaction id with 21 log. Once the transaction has been confirmed, you can check the balance in your bitcoin wallet from the command line with wallet balance, and you can send bitcoin from your wallet to another address with wallet sendto $BITCOIN_ADDRESS --satoshis $SATOSHI_AMOUNT --use-unconfirmed. The --satoshis flag allows you to specify the amount in satoshis; without it the sendto amount is in BTC, but this behavior is deprecated and will be removed soon. The --use-unconfirmed flag ensures that you can send even if you have unconfirmed transactions in your wallet.


Ready to sell your endpoint? Go to slack.21.co

Ready to try out your bitcoin-payable server in the wild? Or simply want to browse and purchase from other bitcoin-enabled servers? Head over to the 21 Developer Community at slack.21.co to join the bitcoin machine-payable marketplace hosted on the 21 peer-to-peer network.