tor2web nginx - check blacklist with lua

30/05/22 - 16h30

I recently operate a tor2web gateway and

it is very recommended to restrict some onion urls.

Why ?

It is very recommended to restrict some onion links. To do this, ahmia.fi provide a list of already known bad hidden services.

To avoid operators to host a list of bad urls, ahmia hash url.onion with md5.

Ok, let's do this, but how ? I didn't find any simple solution so i wrote mine.

How ?

For exemple, nginx has some internal variables in his builtin ngx_http_core_module that we can use to parse requests:


set $hostreq "${host}${request_uri}";

In the code above a variable is setted $hostreq who takes this value "${host}${request_uri}",
respectively the full host (or content of the Host: header) and the uri /something-to-see

The variable ${request_uri} is not important here, the focus is on the variable ${host}

Its is possible to do simple check on the ${host} like this:

if ($host ~* www.target.net) {
    do something;
}

So, I have the amhia blacklist of hidden services in md5 format and i can get the host value submitted to nginx
but how can I read file and compare the ${host} with the content of the blacklist file ?

From nginx directly, I didn't find nothing, but an interesting thing is that nginx has an LUA module ngx_http_lua_module .

Very interesting because it extend the power of Nginx by adding powerful scripting capabilities.

Code !

Now it's time to run with LUA

First I need the blacklist and parse it cause the original form is not one hash per line.


echo $(curl -sLk 'https://ahmia.fi/blacklist/banned/') | tr ' ' '\n' > /etc/nginx/ahmia_onion_blacklist.txt

The list looks like:


0000df9ec834c3b9629711da4688326d

0003cab06fd11db63a23cb59a6e28b37

0007113d199f1907550c2f7a3b8b2e46

0007f81cce5fa0b0ca7c28a3f0712e94

0008f4726e2b9231c09355c0a176cd94

000ae4137fab55b79427ceb4c9c78546

000daa03f6ef54bf0d4e3695eab85cbb

0010e177f6fcaf499b1fd518046bde0a

0013a67e08315e293c274799fa865e01

00140a082312f66c426de5ab0c2c6f1c

...

Ok, globaly what I want is:
1. Read the file /etc/nginx/ahmia_onion_blacklist.txt
2. Store the file content into an array
3. Access to the nginx ${host} variables
4. Get the full onion address (without the tld of the gateway (remember it, it is for a tor2web gateway))
5. Hash the full onion address in md5
6. Compare the hash with the array created from the file
7. Set an nginx variable who will be used later in the nginx conf by proxy_redirect

Let's go !

1. Reading the file is very simple:

local file = io.open("/etc/nginx/ahmia_onion_blacklist.txt", "r");

2. Declare an array then store the file content inside with a for loop

local black_md5_arr = {}
for md5line in file:lines() do
  table.insert (black_md5_arr, md5line);
end

3. According to the doc it is possible to read and write nginx variables.

local church_host = ngx.var.host

It get the host content -> hidden_service_hash.onion.church.

4. What I need to continue is to delete the .church part.

Let's create a new variable and use gsub to do this:

local onion_host = church_host.gsub(church_host, ".church", "")

5. I love hash, so i use the ngx.md5() function:

local onion_digest = ngx.md5(onion_host)

6. This part is the most important.

To read an indexed array in lua the is the iterator ipairs()

for i, md5 in ipairs(black_md5_arr) do
  if md5 == onion_digest then
    ngx.say('<html style="text-align:center;color:orange;background-color:#1E2021"><h1>forbidden access</h1></html>')
  else
    ngx.var.server_id = onion_host
  end                                                                                                                                                             
end

Here i is the element's index in the array (lua's arrays starts with index 1),

and md5 is the element.

7. During the comparison of the url.onion md5 with the array there is two possibilities:

The md5 is in the array then the user is redirected to forbidden message.

Or the md5 is not in the array then an nginx var who takes value of onion_host variable is created who will
be used later in the nginx vhost conf by the proxy_redirect.

ngx.var.server_id = onion_host

Lua's ngx.var.server_id variable will be the nginx's $server_id variable.

(the variable ngx.var.server_id can't be declared as local here.)

The is the full code:

default_type 'text/html'; # nginx directive for html rendering (the forbidden message)
access_by_lua_block {
  local file = io.open("/etc/nginx/ahmia_onion_blacklist.txt", "r");
  local black_md5_arr = {}

  for md5line in file:lines() do
    table.insert (black_md5_arr, md5line);
  end

  local church_host = ngx.var.host
  local onion_host = church_host.gsub(church_host, ".church", "")
  local onion_digest = ngx.md5(onion_host)

  for i, md5 in ipairs(black_md5_arr) do
    if md5 == onion_digest then
      ngx.say('<html style="text-align:center;color:orange;background-color:#1E2021"><h1>forbidden access</h1></html>')
    else
      ngx.var.server_id = onion_host
    end                                                                                                                                                             
  end
}

To finish it is necessary to set a cronjob for keep the blacklist up to date:


*/30 * * * * echo $(curl -sLk 'https://ahmia.fi/blacklist/banned/') | tr ' ' '\n' > /etc/nginx/ahmia_onion_blacklist.txt

Now it is possible to check if the hidden service is blacklisted.

If it is, the gateway refuses to display the onion content to display an error message.

i@mn0t.dev:~#

Why ?

How ?

Code !