Python

After or rather while following the Udemy’s ‘2020 Complete Python bootcamp: From Zero to Hero’ by Jose Portilla (Hi!) I decided that I could rewrite the perl script into python. And so I did. Below is the version that resulted from that effort. It can surely be smarter, so poke me on my email address if that is possible and I’ll update this. Thanks Jose for your great course! I appreciate it!

The script

#!/usr/bin/env python3
#

## 25/09/2020:
## Python based E.T parser by Remko Lodder <remko@elvandar.org>
##
## The script is based on the original perl script by Joshua Gimer and an unknown author who created
## my reference version: https://doc.emergingthreats.net/pub/Main/EmergingFirewallRules/emerging-ipset-update.pl.txt
##
## The netaddr functionality in the form of IPNetwork and cidr merge
## are taken from the website: http://www.korznikov.com/2014/08/creating-black-list-of-ips-for-iptables.html
## Thank you for the pointers there, which I shamelessly used to create this variant.
##
## The script fetches the ip addreses/ranges that are potential treats and creates an ipset list from it.
## The ipset list is then used by iptables to produce a working firewall set.

import time
import urllib.request
import os
import re
import syslog

from netaddr import *
from socket import timeout
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError

# Prototype variables
n = False

# Bootup messaging and syslog
syslog.openlog(logoption=syslog.LOG_PID, facility=syslog.LOG_INFO)
syslog.syslog('Starting Emerging Threats (ET) IPTables update script....')
print ('Starting Emerging Threats (ET) IPTables update script....')

# Two times a day timer
timer = 43200

# Sleep a bit after an timeout.
timeout_timer = 120

# The location of the Emerging Threats revison number file.
emerging_root = 'https://rules.emergingthreats.net/fwrules'
emerging_fwrev = emerging_root + '/FWrev'
emerging_fwrules = emerging_root + '/emerging-Block-IPs.txt'

# Temporary files
tmp_dir = '/tmp'
rules_file = tmp_dir + '/emerging_iptables2.txt'

# Binary location
iptables = '/sbin/iptables'
ipset = '/sbin/ipset'

# Iptable chains
iptables_att_chain = 'ATTACKERS'
iptables_drop_chain = 'ETLOGDROP'

# ipset names
ipset_botccnet = 'botccnet'

# Get the current IPTables ruleset revison number.
def get_fw_rev():
    response = urllib.request.urlopen(emerging_fwrev)
    data = response.read()
    text = data.decode('utf-8')
    return text

# Get the firewall rules and ignore the errors
def get_fw_rules():
    try:
      with urllib.request.urlopen(emerging_fwrules) as response, open(rules_file, 'wb') as out_file:
            data = response.read() # a `bytes` object
            out_file.write(data)
    except HTTPError as e:
        print('There was an error: ', e.code)
        pass
    except URLError as e:
        print('Something went wrong in reaching the server: ', e.reason)
        pass
    except ConnectionResetError:
        print('---> Connection reset, retrying in ' + timeout_timer)
        time.sleep (timeout_timer)
        process_et_rules()
        pass
    except timeout:
        print('---> Connection timed out, retrying in ' + timeout_timer)
        time.sleep (timeout_timer)
        process_et_rules()
        pass

# Be able to fork a process
def parent_child():
    n = os.fork()

    # If N is >0 then a child had not yet been forked (we are in master process)
    if n > 0:

        # Setup first tables, they will not be readded later on.
        os.system(iptables + ' -N ' + iptables_att_chain)
        os.system(iptables + ' -N ' + iptables_drop_chain)

        # Flush previously assigned iptables and ipset parameters
        os.system(iptables + ' -F ' + iptables_drop_chain)
        os.system(iptables + ' -F ' + iptables_att_chain)
        os.system(iptables + ' -D FORWARD -j ' + iptables_att_chain)
        os.system(iptables + ' -D INPUT -j ' + iptables_att_chain)

        # Create new iptables and ipset parameters
        os.system(iptables + ' -I FORWARD 1 -j ' + iptables_att_chain)
        os.system(iptables + ' -I INPUT 1 -j ' + iptables_att_chain)
        os.system(iptables + ' -A ' + iptables_drop_chain + ' -j LOG --log-prefix "ET BLOCK: "')
        os.system(iptables + ' -A ' + iptables_drop_chain + ' -j DROP')

        # Remove current ipset list, and recreate it.
        os.system(ipset + ' -X ' + ipset_botccnet)
        os.system(ipset + ' -N ' + ipset_botccnet + ' nethash')

        # Create the ipset matching chain
        os.system(iptables + ' -A ' + iptables_att_chain + ' -p ALL -m set --match-set ' + ipset_botccnet + ' src,src -j ' + iptables_drop_chain)
        os.system(iptables + ' -A ' + iptables_att_chain + ' -p ALL -m set --match-set ' + ipset_botccnet + ' dst,dst -j ' + iptables_drop_chain)

    # else if N = 0 then that means that we are in child mode and can start processing the et rules.
    else:
        process_et_rules()

def process_et_rules():
    # Reset revision number before starting.
    ip_list = []
    rev_num = 0

    while True:
        old_rev_num = rev_num
        rev_num = get_fw_rev()

        if int(rev_num) > int(old_rev_num):
            get_fw_rules()

            # loop through rules file, remove empty lines and process the remaining ip
            # by using a regular expression that matches <wordboundary>ipaddress<wordboundary>
            with open(rules_file, "r") as read_file:
                lines = [line for line in read_file.readlines() if line.strip()]
                for line in lines:
                    line = line.strip()
                    if re.findall(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\/?[0-9]{1,2}?\b',line):
                        ip_list += [IPNetwork(line)]
            ip_list = cidr_merge(ip_list)
            amount_addresses = len(ip_list)

            # Flush entries in the ipset list.
            os.system(ipset + ' flush ' +  ipset_botccnet)

            for ip_address in ip_list:
                # Add new entries from the list.
                os.system(ipset + ' -A ' +  ipset_botccnet + ' ' + str(ip_address))

            # Summarize what we did, so that we have an idea how many addresses should be there.
            syslog.syslog("Wrote %d addresses to the ipset list, using ET version %d, going to sleep...." % (amount_addresses,int(rev_num)))
            time.sleep (timer)
        else:
            syslog.syslog("The old version: %d is the same as the current version: %d... sleeping a bit and retry...." % (int(old_rev_num), int(rev_num)))
            time.sleep (timer)

# If a child is not yet found, fork
if n == 0:
    parent_child()

So, since the beginning of october we decided to inform our customers and ask them to move to another provider at the end of the contract. All our customers migrated quickly and recently we migrated away the last customer. That marked the end of the hosting servers, we have them on shutdown and they will be removed in a little. The site is still available as placeholder on my host and mail for the domain will still be incoming for a little more.

For our customers and interested parties: Thank you for the trust in the past 15 years! It had been a blast.

One day, we will meet!

Image used from express.co.uk and referenced from them:

To quickly summarize the day: It was great but exhausting :-).. numberous talks with coworkers but also old friends (Hi Johan, Rene, Ronny, Alain, Cor, I mean you for example!). For me this was a first time as technical field manager at a conference. The SUE team was the biggest team of them all I think so that was great to see. Thank you all coworkers for your visit and your big enthusiasm. I hope that we can visit many more of these conferences!

About the talks:

The keynote (by David Blank-Edelman) at first was very interesting and eye-opening. Not only because of the talk, but also because of the employer of David, Microsoft. A while ago I found that MS was seen more and more on open-source territory and I think this demonstrates that I am right about that. Nevertheless we got a serious talk about Site Reliability Engineering and what angles you can look at it. The main thing that David demonstrated is that “it depends”. You need to find a way to first describe your company and wishes, but also your clients. Reliability in this regard is measured at the client instead of internal monitoring/metric gathering. For example if you have 100 servers and a few of them go down, are you in panic ? It depends! If the customer doesn’t notice anything, just keep chilling. If the client does notice anything, like not being able to access your site, or add items to the shopping cart.. then make sure it starts working again! You need to define SLI and SLO’s (Service Level indicators, which metrics at what “item” in the business chain and Service Level Objectives: How do you want things to perform, what is your goal). I think you can do that with proper setup monitoring that not only checks availability but also does an actual login, or an actual shopping cart experience. When I worked for a major ISP in .NL they did that with Selenium. The setup replicated a user login from various remote places.

My next talk was about making scripts better by my old coworker and friend Michael Boelen. I enjoyed this talk, I am experienced in writing shell scripts, but still I learned a few things that I didnt know before. I also understand better on how to approach creating new scripts and what the caveats are for being posix compliant. Michael had the crowd on his hand, he got a very interactive talk and still keeping track on the objective from his talk. It was appreciated Michael!

After that I visited the talk about “treating documentation as code” by Hagen Bauer. I visited that with in my huge documentation experience from FreeBSD. Hagen has a setup where he uses asciidoctor and some modifications to print his documentation in various formats. It can use external input as well as just regular ‘md’ kind of files. So once prepared, you can more easily write documentation just in ascci/plain text and if you have set it up with a CI/CD pipeline for example you can auto generate new documentation when you have a commit or change in your GIT repo. I think this lowers the barrier if there is a template and all you need to do is fill in an text file (With some markup). If I ever have time, this is really interesting to understand better!

After Hagen’s talk we had lunch. There were many things to choose from, a dangerous approach because of an after lunch dip risk.

Next up after the lunch was a talk from Koen de Jonge, board member for NLUUG. His talk was about a dream or idea: Community Hosted Open Source Infrastructure. (CHOSI.org). This dream started with how we (I include myself there) used to learn and do things, touch real hardware, modify kernels, wait ages for an kernel or program compilation completed and noticing that you made a mistake. Where nowadays people “take care” of you. The educational value of having in-depth knowledge about products is going away. The cloud offers items that you dont have to touch at all. Ofcourse the cloud infra people need to do so, but that group of people is being reduced in the world. See for example RHCE8, you learn ansible.. which is a great and fun thing to do, but you dont learn in-depth technical hacks on the commandline anymore. With this talk Koen tried to take the audience to a world which we knew from the past and bring that back. The general idea is to have at least one or more racks with own equipment which you can use to start all kinds of vm’s, from linux to a bsd to solaris. With a “bierviltje” calculation he noticed the required funding which would be approx 12 euro per user. I am very much interested!

Further down the road I visited the XS4all moet blijven talk from Anco Scholte ter Horst, current CEO of “Freedom Internet” (The new XS4all). Anco took us down the road which was followed after announcing that XS4all needs to be assimilated by KPN. He told us about the fight they have put up to save the company, to see alternatives and finally after another reply from KPN that they were going to assimilate XS4all and drop the company behind it, the birth of Freedom Internet. A very nice and driven talk from Anco, the crowd was very interested including myself. Lets see where this is heading and hopefully they can do what they want to do!

Next in line I visited “what does vNUMA actually mean?” by Wim ten Have. Wim appears to be extremely technical and possesses knowledge that not many people have. During this track I was from time to time very lost. Not because Wim didn’t explain, but because I didn’t cope the in depth knowledge. NUMA is known for making processors able to directly access memory regions. In order to use that effectively you should combine CPU’s that are near each other and share the same NUMA domain. vNUMA is an addition to QEMU / KVM and enables automatic mapping of NUMA within a VM basically. (Someone correct me if I am wrong ;-)). Sadly, if you are not that experienced you will loose much of the information presented.

Just before attending the last talk, I joined Martin Geusebroek’s talk about “Counter Social Engineering”. Martin is an experienced HUMINT officer and extremely knowledgeable about this subject. He gave multiple examples on how things work. There were a few demo’s / recordings of social engineering stuff that actually worked. Next to that he also did a “remember the given names”, where he did try to influence your brain. In the recap we were presented the given names. One of them was not in the list but your head thought it was, because a lot of the words shared the same topic. Your brain fills in the details. That makes social engineers able to influence you without you actually realising. When I worked at ING we had such trainings periodically and then for entire days. Always think who asks what and why. Try to properly verify someone’s identity and when in doubt, just get help from senior management. Even if the director is giving you a hard time, it’s in his companies best interest if you are very firm and solid in your work. Thanks Martin for bringing this topic to NLUUG!

and finally the closing keynote: Tales (Fails) from the trenches… by Edwin den Andel. Edwin is a very classic hacker, in the sense that the name hacker was actually ment. Edwin is creative and thinks out of the box to try and obtain information. Not for the worse but for the better. Nowadays hackers are seen as nerds that break into computers and steal stuff. Edwin and Zerocopter behind him try to address that. They want to receive vulnerability information and highly suggest that you do not download entire datasets, but just one or two rows to proof that you can access data. Else you will get into the dark mazes of the law and you might even be prosecuted. Next to advocating the right thing, Edwin also gave numberous examples on how companies failed. I felt really connected with this topic and I think a lot of people found this the best talk of the day. Edwin easily presents his knowledge and is easy to follow. Edwin, I enjoyed your talk a lot, thank you!

After all these talks people needed drinks and beverages. My employer SUE sponsored these and a lot of people stayed until it was time to wrap up. Together with a few coworkers from SUE, we were the last ones to leave the conference. I hope to be able to rejoin the NLUUG conference next year, either in my current role or in a new role.

In het bedrijfsleven, werkende voor Sue, ben ik dus gewend dat er een goede strikte focus is op ontwikkeling en opleiding.

Wij verzorgen regelmatig opleidingen in-house, en geven die of zelf met onze gecertificeerde collega’s of er komt een leraar extern binnen om de opleiding te geven. De groepen worden niet te groot gemaakt zodat een ieder de benodigde aandacht kan krijgen. Tijdens onze studie dagen/weken is het mogelijk om savonds te blijven mee eten en met elkaar kennis te kunnen delen. Wij staan dus voor een kwalitatief hoogwaardig leertraject.

Mijn vrouw is leerkracht, een werkgebied waar ze voor gekozen heeft om onze toekomst (de mensen die ons straks moeten onderhouden!) te begeleiden en op te kunnen leiden zodat ze een fijne periode in de maatschappij kunnen hebben, hun steentje kunnen bijdragen en goed opgeleid ten tonele komen.

Ik had verwacht dat in het onderwijs, waar de dienst “het opleiden van onze toekomst” is, een veel grotere focus heeft dan het commerciele bedrijfsleven. Helaas is niets minder waar. Er zijn heel veel administratieve handelingen. Er komen steeds meer leerlingen in een groep, waardoor persoonlijke aandacht bijna niet realiseerbaar is. Er moeten ontzettend veel zaken buiten schooltijd worden gedaan en geregeld. Ouders worden steeds mondiger en verwachten meer en meer van de leraar, ze worden zelfs boos als het kind in kwestie niet op het niveau scoort die ze graag zien. Door al die zaken is het in mijn ogen ontzettend moeilijk om kwalitatief hoogwaardig onderwijs te kunnen leveren.

Ondanks de grote hoeveelheid werk en taken, is er een enorme onderwaarding voor het werk, een fulltime docent werkt in mijn ogen minstens 50% buiten schooltijd (geen lestijd!) extra. We hebben het dan over dik 60 uur per week. In het bedrijfsleven zouden deze leraren daar een goed salaris en voorzieningen voor krijgen (het is immers de primaire dienst). In plaats daarvan worden leraren beloond met meer werk, meer administratie, en meer kinderen in de klas.

Als wij straks zelf oud zijn, en we kunnen niet de aandacht geven die nodig is, wie is er dan nog goed genoeg opgeleid om ons te verzorgen? Is het echt nodig dat we hierover twijfelen en ons zorgen maken? Niet als er een juiste en gelijkwaardige waardering voor het werk is.

Ik heb ontzettend veel respect voor de leraren die gestaakt hebben en acties opzetten om hier aandacht voor te vragen. Ja het is soms niet zo handig als je kind thuis zit (geldt net zo goed voor ons), maar het is het waard. Het gaat over onze toekomst! De overheid en samenleving moet dit goed regelen, niet 1 malig maar structureel. Ik hoop dat wij daarmee ook zorgeloos van onze oude dag kunnen genieten.

Dank, Remko

How did we get there?

Short summary: I did not feel at home anymore.

Larger summary: The creation of the Code of Conduct within FreeBSD made me frown a lot, and still does. It’s largely American oriented and does not take non-American stuff in consideration, or not enough. The current leadership is more worried about personal social media posts and how to respond to that then about guiding the project into the next phase. The world is not entirely American and different people with different cultures were welcome within FreeBSD. My personal feeling is that that is no longer the case.

I realise that if you read this, this might make you frown as well. I am a long standing community member, which covers a large part of my adult life. Does all this outweigh my long-term connection to the project? Yes.

Beyond the “social” side of the project, I also think that while being conservative, we missed the boat on multiple occassions. Things come in late, or are not “addressed” at all. Take containers. They are the current hype for microservices. There is no way to do something with that within FreeBSD. FreeBSD has jails, which is a more heavy weight container-kind-of-solution. Or better said it is a more lightweight virtual machine instead.

Tools that use containers, like Gitlab CI/CD and many other things make use of those services. FreeBSD just does not have them. It’s not sexy enough to run it in your DC. Sadly I do not see much activity company wise in the Netherlands either that suggests that I am wrong. Most things that I do see in my professional life are Linux related machines.

Is this the end for me?

With my current FreeBSD implementations, yes. All my machines are migrated to Linux, there are no exceptions anymore. This makes it easier for my automation tooling, because everything runs on the same foundation and files can be found on the same place. Same goes for packages etc.

Farewell FreeBSD, you have served me well and I think that I earned the right to use it by all my contributions. I hope that a less politically minded core team stands up at some point and changes the game. Perhaps that will make me rejoin the project that I once was so proud of.

So when I saw Cor at the NLUUG a few days ago.. I missed Paul ofcourse. I had not seen Cor for a few years and not after Paul’s passing. Seeing Cor alone instantly reminded me of Paul. You are still missed Paul. Rest in peace!

Image taken from db.net where both Paul (Left) and Cor (Right) appeared on photo.

History

Due to various reasons not discussed here, I was not able to see to a proper translation. Rene did a lot of work (thank you for that Rene!).

The PO system

First of all, i am going to discuss a bit about the PO system, which is a gettext way of doing translations. It chops texts into msgstr’s (message strings) and then translates those strings using msgid’s. Same lines are translated the same, this might be a good option, unless the context changed between the lines and then you might get ‘google translate’ kind of ways.

Back to the story…

After getting time again to see this through I noticed that we started using the “PO” system, using gettext. Our handbook (for example) is now translated into one huge book.xml file which is then cut into msgstr’s that can be translated to msgid’s. For this I use the poedit application (the PRO version) so that I have counters and translation suggestions from the online Translation Memory(TM) that we all develop. I also contribute the FreeBSD translations back to the TM so that everyone can profit from it.

I am now first synchronising the Glossary because that didn’t change much with the current online translation and working my way back to what had been translated already and translating the missing bits and pieces in between. Mike (co worker at Snow) also did a tremendous job in getting this into better shape the last year which had not yet been merged back to the online variants because it was not yet complete. I can use that information though to generate a manual handbook variant of that version and use that to even further use the current translation effort into the gettext/po system.

Biting the bullet

As one of the first translation teams to use this, I expect to hit some rocks on the road. For example, there are lines that do not need translation, mailing list names are the same in every language, perhaps the description changes but not the ‘realnames’. Same goes for my entity (&a.remko) which does not change, nor my PGPkey. And if those things change, they require changing over all translation efforts as well as the original english version. We are looking into a way to ‘ignore’ them for the po system but include them when building. So that pgpkeys and such are always up to date.

I also had been discussing this with Vaclav the developer of poedit, and he mentioned that it does not matter much, because when a line changes and you update the po, those lines will be invalidated and need ‘retranslation’ for the entire string. So that all gets us in interesting situations that we did not encounter before. I am biting the bullet myself after we have discussed this a few years ago and I hope that the entire project can benefit from that.

Alternative options, pre-translate, merge current translations automatically?

And yes, a valid question would be, cannot you merge the current translated information into the po system automatically. If every word was on the exact same spot and line, yes this might be an option. Sadly because of grammer and different wording (longer/shorter) this changes rapidly from line 1 already and is thus not easily done. If you have suggestions however, we are always willing to listen. Please join us on translators@FreeBSD.org so that we can discuss those things better :-).

I started debugging with Ubiquiti ages ago to see where the problem lies. In the meantime I was unable to continue with this, but I had an interesting thought yesterday. I sniffed the traffic from the hardware (mac) address of the ereader and noticed that it tried to resolve: http://www.msftncsi.com and fetch /ncsi.txt. The site is a microsoft network connection information page that informs microsoft systems whether or not an active internet connection is seen.

Somehow it seems that Kobo is also using that for it’s android based readers as well. Without it, the network connection just disconnects and does nothing. That is somewhat upsetting because the device is just perfectly able to connect to the network(s) and has relative free internet access. One thing is that I filter on DNS responses and exclude known malware/spam hosts and analytics sites like google. This reduces the amount of advertorials on the internet and bogus trackers. It seems that msftncsi.com is also on that list and thus gets an NXDOMAIN when querying for it.

I do not entirely understand why an ereader would need this kind of information before being able to connect to the internet. The device should associate with a WiFi access point and get an address and the like. Whether or not that gives continued access to the internet is something that is a next step. So instead of giving up, it could just mark the WiFi symbol with an exclamation mark (!) to report that something might not work and/or just try to connect to the kobo internet environment. That would be more common use of the internet then depending on an internet file which might be blocked (such as in my case).

For now I changed my caching mikrotik’s to include msftncsi.com as a static entry and point that to my webserver and service the file instead. That makes sure the Kobo can connect to the environment and gives me full access over that file instead of some bogus remote site that might do nasty things (without me knowing).

Ofcourse I asked (nice and polite) Kobo to change this interesting behaviour.