TheHackersManual2015RevisedEdition

Bringing the two together

These logical addresses work by mapping to the physical MAC addresses that we covered earlier, with each IP address being linked to a particular MAC address. You can see this on your own computer by using the ip neighbour command in the terminal:

ip neigh show

Each line output by this command represents one host that is reachable by your computer. The first entry is the host’s own IP address, while the second two specify which NIC it is reachable through, and the lladdr specifies the MAC address of the device. Your computer can then use this information to construct packets that are addressed for the correct computer.

Now, here’s the clever bit: computers that have the same network address store this information about each other and can communicate directly. If they don’t have the same network address, then they don’t store this mapping of IP to MAC addresses, and so can’t communicate directly. Instead, their communication happens via an intermediary, known as a gateway or router (see p112 for how to hack your own router and take control). Routers keep track of all the IP to MAC address mappings for their own network, but they also keep track of how to communicate with other gateway routers and networks.

If a host on its own network wants to communicate with a host on another network, it sends its message to the gateway, with the full IP address, and then the gateway sends this message on to the appropriate network. The gateway at the other end will then route the

packet on to the correct host on its own network.

If these other networks have hundreds, thousands or even tens of thousands of hosts on them, you can see

how this reduces the amount of information each host or router has to store. Instead of storing the MAC addresses for all these hosts, it only needs to concern itself with one address on each network – the gateway’s address.

Investigating routing information

The knowledge of what path to send packets via – for example, from Computer A to Router A, and then from Router A to Router B, before finally reaching Computer B – is known as routing information, because it’s information about the route the packet must take.

You can see your computer’s own routing table by using the route argument of the ip command:

ip route show

The first line of this output says ‘default’ before specifying an IP address that packets should travel via, along with which

interface the packets should be sent out of. This default route is your computer’s fallback – if there’s not another route specified for the IP address you’re trying to communicate with it will send packets to this gateway.

If you compare the destination of the default route to the output of the ip neighbour command, you’ll see that your computer has the MAC address of its default gateway, and so the two are able to communicate directly.

Arping around

Your computer builds the table of MAC to IP address shown in the ip neigh command by using the Address Resolution Protocol (ARP). This protocol defines a special packet, just like the ones we saw above when we ran the tcpdump command. When your computer wants to find out the MAC address of a computer with a particular IP address, it will

construct an ARP request packet.

This includes your MAC address as the sender, and the IP address of the host whose MAC address you want to know, but it doesn’t

specify the destination MAC address. This packet is then broadcast to every host connected to the same physical LAN as you – that is, it will be sent out of every port on a switch, but not from one router to another – saying Request who has 192.168.1.2 tell 192.168.1.1, for example.

If that IP address exists on the network, a REPLY ARP packet will be sent back to the requester’s MAC address, saying Reply 192.168.1.2 is-at ea:34:43:81:02:7c. Now the two computers will be able to communicate directly.

You can see this in action by running the tcpdump command from earlier again. Instead of opening a web page, however, in another terminal, run as root

arping 192.168.1.1

where the IP address is that of your default gateway. If you now look at the tcpdump output, you should see an ARP conversation taking place. Θ

Switches are the unsung core of many networks,

reducing collision domains and making it possible for many hosts to communicate reliably in a single network.

In February 2011, IANA (the Internet Assigned Numbers Authority) distributed the five remaining /8 blocks (each comprising 16 million addresses, and being the scrapings

of the recovered address barrel) among the five Regional Internet Registries (RIRs). Now they have nothing left to give, they are gone. Item discontinued. Sure, you'll still be able to get IP addresses for new devices and virtual machines and what not, but companies' supplies will go dry and there

ain't gonna be no re-up.

There's no need to panic, though – many years of thought and testing have already come up with a replacement addressing protocol, and it's been considered productionready in Linux since 2005. Some major players have already activated their IPv6 networks. If you're in the UK, the chances are your ISP doesn't yet provide you with IPv6 connectivity (a handful do, and those lucky enough have no need of this tutorial), but we can work around this using a tunnel later. It's important to note that there is no plan to pull the plug on IPv4

– the two protocols can work perfectly well alongside each other. Network adaptors can be assigned both types of addresses, and dual-stack routers can marshall both types of traffic. But post-exhaustion, new hosts will only be accessible by IPv6, so those with only IPv4 connectivity will find

themselves no longer able to access parts of the internet (and to some extent vice versa). While some network hardware (such as routers) may need to be upgraded, bread- and-butter network cards and switches will work fine for IPv6

– all they care about is transmitting Ethernet frames, indifferent as to the nature of the higher level packetry contained therein.

IPv4 uses 32 bits to address nodes, with the convention being to divide into four groups of eight bits (a byte, or octet) and write the decimal representation of each octet separated by dots. Thus IPv4 address space allows for about 4.3 billion addresses, which coincidentally is about one for each person on the earth who enjoys internet access. As more and more devices connect to the internet, and carrier-grade NAT address sharing regimes are not seen as an appropriate solution, we rapidly approach IPv4 exhaustion. IPv6 addresses, by comparison, use 128 bits for each address, which means that we won't be running out any time soon (there's enough for each atom on the surface of the earth to get one). The standard notation is to partition an address into eight groups of 16 bits. Each group is written as four hex digits, and separated from the next by a colon.

Because 32 hex digits, and seven colons, are cumbersome to write down, there’s a couple of shortcuts. Shortcut the first: any leading zeroes can be omitted from each group, so 0123 can be written 123. Shortcut the second: one series of adjacent 0 (short for 0000) groups can be replaced by ::. As an example, consider the loopback address (the analog of 127.0.0.1 in IPv4) which, when abbreviated by the first rule, is 0:0:0:0:0:0:0:1. Those first seven groups can be unceremoniously discarded and we need only write ::1. Note that you can only use the double colon notation once in an address, because multiple applications would be ambiguous. An IPv6 address is divided into two parts: the first 64 bits (or four groups) form the network prefix and the last 64 are termed the host identifier. The network prefix is further divided into a routing prefix and a subnet ID, but we’re not going to worry about that here.

If you're using an up-to-date distro, it'll probably have IPv6 support built in. You can check this using the ip command from the iproute2 package:

$ ip a

...

2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

link/ether 90:2b:34:aa:bb:cc brd ff:ff:ff:ff:ff:ff

inet 192.168.1.144/24 brd 192.168.1.255 scope global enp5s0

IPv6 uptake has, on the whole been notoriously slow, despite many large firms having flipped the appropriate switches and IPv6 being mandatory for the 4G mobile standard.

Part of the reason for this is that IPv4 has proven itself to be something of a champion. The IPSec and DNSSec security extensions, which were originally designed as part of IPv6, were incorporated into it, which made for one less reason to switch over. In 2014, though, many older corporate routers began to baulk as BGP tables exceeded 512K entries. It is speculated that this was the cause of downtime at eBay and Lastpass. While everyone recovered reasonably

swiftly – a few auctions notwithstanding – this provided something of a wake up call to everyone concerned.

Parts of the system were beginning to creak somewhat ominously, and network bods began to act. As a result, you will find that, if you turn off IPv4 traffic, much of the internet still works as you would expect.

If you're using a Teredo tunnel, and have set it up to communicate on UDP port 3544, then the following iptables rules will do a good job of blocking any other outgoing requests, effectively cutting you off from the IPv4 internet, except for your DNS server:

#iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#iptables -A OUTPUT -p UDP --dport 53 -j ACCEPT

#iptables -A OUTPUT -p UDP --sport 3544 -j ACCEPT

#iptables -A OUTPUT -j REJECT

You'll find some sites work as expected, some

kind of work, but most don't. You probably need IPv4 connectivity, so once you've finished experimenting, you should delete these rules. If you didn't have any to begin with, you can flush the whole lot with:

# iptables -F

As an alternative to using a tunnel broker for a single machine, it is possible to use a Teredo tunnel. Teredo is a Microsoft-designed technique, which relies on the bandwidth of so many Teredo relays. These act as gateways, unencapsulating your IPv6 packets and sending them off, and wrapping them in IPv4 packets for return. It is intended as a last-resort technique, and because only one IPv6 address can be assigned per tunnel, it is unsuitable for use on networks, even small ones. Because it creates a tunnel that can happily sidestep your firewall, it is widely seen as

an obvious security risk. In general, its use tends to be discouraged and it’s being phased out. It is, however, very good at traversing NAT layers (teredo is actually a genus of creatures that bore holes through ship’s hulls), so it’s handy if your external IPv4 address changes regularly or your network is otherwise unco-operative.

An open source client by the name of Miredo is available in your distribution’s repositories. Once you’ve installed it and started the service (systemctl start miredo for SystemD), you should be enjoying IPv6. Toredo addresses

have a reserved /32 prefix, and all Toredo addresses have 2001:0: at the beginning. If all went as planned, you should have been assigned one. You can test this out by visiting http://testIPv6.com which will give you (or at least your IPv6 connectivity) a rating out of 10. You aren’t ever going to get full marks with a Teredo connection, and it’s also possible that the Miredo service got your internal IP wrong. In the latter event, you simply need to edit /etc/miredo/ miredo.conf and add a line such as:

BindAddress 192.168.1.10

Logs give you mountains of useful information: don’t throw it away, or risk losing it.

Syslog is a venerable part of your operating system, whatever flavour of Unix you run, tirelessly collecting information about system and

application events in text files under /var/log. Whether you are administering thousands of servers, or just looking after a simple home network, ‘check the log’ should be one of your first thoughts whenever a problem occurs.

We’re going to build a logging server, and take a quick tour of how to tell your other devices to send their logs there, but first let’s take a look at why it is such a good idea to stick your logs on another server.

Logs are an admin’s best friend. They tell you: what the

Logging daemons

We’ve briefly mentioned security. You should read the logs on a compromised box but, at best, you can’t wholly trust their completeness. Export your logs to a server that’s only open to receive them on one port, only SSH-accessible from one place, and has been reasonably hardened, and those logs become more trustworthy. Whether you use a security hardened distro, or plunder the Linux Format archive for security tutorials, or just trust to luck, we leave it to you.

Physical back-ups/archives, moved off-site, are the finishing touch to an arrangement which should help you sleep soundly at night. Think this is over the top? Well, on a home network you might simplify (or even omit) the archiving, but a separate box – even a Raspberry Pi with a couple of redundant old USB hard disks – will provide a permanent simplification to any future logging problems.

In this tutorial we’ll be using Rsyslogd, Rainer Gerhards’ improved version of Syslogd. This has benefitted from competition from Syslog-ng, a freemium product that has gradually moved advanced features from the proprietary to the free version. Rsyslogd is entirely free and open source, however, and the default logging daemon on almost every distro. It’s also slightly easier to configure, though not enough to stop you taking a look at the alternative, should there be a feature in Syslog-ng that you need that hasn’t appeared

in Rsyslogd.

We will show you the basics of setting up a remote logging server, getting a client to send its logs there, and looking at Apache as an example of a service with its own logging, that

:msg, contains, “user nagios” ~

discards Nagios messages. Sometimes you’ll want to discard something during setup, and sometimes you’ll run a program that fills logs with information that you’ll never need. Note lines are actioned from left to right, so general rules should be put first, with specific exceptions and additions after the comma where necessary.

Try a few changes on your rsyslog.conf file, just to get familiar with the format, before we start on remote listening. To get Rsyslogd to read your config changes run:

kill -HUP (cat /var/run/rsyslogd.pid)

You can test your rsyslog.conf changes from the command line with logger, for example:

logger -p local6.warn “This is a test through logger.”

Listening for traffic

Time to skip back to the earlier rsyslog.conf directives we ignored before. First off, lets deal with Modules. You should have the default of

$ModLoad imuxsock $ModLoad imklog

which provides support for local system logging and kernel logging respectively. To get your server listening for remote traffic via either UDP or TCP you’ll need to load their respective modules, too:

#provides UDP syslog reception $ModLoad imudp $UDPServerRun 514

#provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514

The port number is up to you, of course, but a lot of boxes

are preset for the standard 514. UDP is stateless, meaning client machines will continue to send log files even if your log server is too overloaded to receive all of them. TCP ensures everything should get through, but at the expense of increased load.

Next, the Global Directives section deals with file ownership and permissions, but also enables you to specify a subdirectory for application-specific conf files: $IncludeConfig /etc/rsyslog.d/*.conf

Ubuntu does this by default, and your server may include files such as /etc/rsyslog.d/postfix.conf, for example, with a line to create an additional socket in postfix’s chroot in order not to break mail logging when rsyslog is restarted: $AddUnixListenSocket /var/spool/postfix/dev/log

Now, ahead of the other rules, you can start sending your logs to the remote log server. Our example server is on the local network, at 192.168.0.2, listening on the standard port, so we can send our logs there with:

*.* @192.168.0.2:514

A second @, for example @@192.168.0.2:514 will send the logs via TCP instead of UDP. You are unlikely to want *.* from every client, however.

We’ve now seen rsyslog.conf set to local files, remote servers, and piped to named streams (/dev/console).

You may also want to send a log to a particular user (eg root) or even all users:

kern.* root *.emerg *

but also useful is logging to one particular console: auth,kern.* /dev/tty2

This example lets you know any issues with the kernel, and who’s logging you on, and you can leave a monitor plugged in to a server and just check when you wander by or use a KVM. A quick poke around /var/log on a typical server reveals many services logging to their own files or directories, without any help from (r)syslog. In most cases other services can be

persuaded to use syslog, greatly simplifying your shifting of the log files to the remote server. For example, in Apache edit the config file in /etc/apache2/conf.d (under /etc/httpd on Red Hat-based distros) by commenting out all default logging directives, and adding one which calls the BSD utility logger, the shell interface to syslogd:

ErrorLog “|/usr/bin/logger -p local6.error” combined

All *nix systems store their many log files under /var/log, and it’s worth becoming familiar with the most important. Note, however, that on Ubuntu and Debian-based systems there is some blurring between /var/log/syslog and var/log/messages, but broadly speaking the later should capture all the important but noncritical messages, while the former folder logs nearly everything except auth messages. (You’ll need to look in auth.log for those.)

/var/log/dmesg

This is the kernel ring buffer, which is also available via the dmesg command.

/var/log/auth.log

/var/log/Xorg.0.log

This the X startup log file. The source of information for problems with your X config.

/var/log/btmp

A binary file containing attempted bad logins to the system, accessed via the lastb command.

/var/log/wtmp

A binary file that logs all users who’ve logged in or out, which is accessed via the last command.

/var/log/daemon.log

This file supplies detailed Information about the running system and application daemons.

CustomLog “|/usr/bin/logger -t apache -i -p local6.info” combined

Combined is the format. Custom formats can be specified through Apache’s LogFormat directive. local6 is one of the eight local facilities under syslog reserved for local use. There is no defined standard, but typically PostgreSQL uses local0 and SpamAssassin uses local3.

Now we need to get these local6 logs sent off to the remote logging server. You may want to keep a handy local copy of the error logs in /var/log/apache/ or even with Apache’s local files:

local6.* @192.168.0.2:514 local6.error /etc/apache2/logs/error_log

On the logging server, you can now route incoming local6 logs to their own file:

local6.* /var/log/apache_logs

Don’t forget to restart Apache and reload the syslog.conf file on both remote and local machines in order for your changes to take effect.

Monitoring logs

As we’ve discovered, logs tell us useful things, and searching logs tells us useful things that have happened in the past, but monitoring logs lets us know what’s happening now. For instance, whether someone is trying a buffer overflow attack on your web server. A simple Perl utility like Swatch can be set to email you each time an extremely long URL is called, for example. You do this by configuring its conf file with a regular expression to look out for in the logs, or perform any other action you wish, even tweeting if you have a command line Twitter app installed and configured:

# sample ~/.swatchrc

watchfor /File name too long/ ./ttytter -status=”Buffer- overflow attack in progress on my Web server. #LeaveMeAlone”

Logwatch is another old skool Perl script for keeping an eye on your logs. It is designed to be run from a remote machine or non-writable medium, as easily as it would be to run locally. Its security, then, can be good but first you’ll need to learn how to write your own filter scripts for Logwatch. You’ll also need knowledge of Regular Expressions if you use

multitail, which shows you multiple logs, filtered and colourhighlighted, in multiple windows.

If this sort of configuration effort isn’t your cup of tea, there are alternatives: from enterprise-friendly Java-based data analytics systems, such as Graylog2 – which can manage billions of events and process hundreds of thousands of new events every second – to Intrusion Detection Systems (IDS) that combine log analysis with file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response, such as OSSEC. Even a brief roundup is beyond the scope of this tutorial, but we suggest that, unless you’re managing a data centre or large enterprise set-up, get the simplest solution that meets your needs.

Management

Now you’re collecting all these logs, you need to manage them. Logrotate is the established Linux utility for handling compression and purging of log files. A good look through /etc/logrotate.conf and the individual scripts under /etc/ logrotate.d will show you some sensible defaults for a standalone machine. Some tweaking could give you a workable setup for your logging server, but most likely you’ll need to monitor disk use and log growth for a while, and continue to do so as you add more clients to your network

to get a satisfactory balance between disk space, and keeping enough logs for peace of mind.

Compliance issues for larger companies, and simple agreements on what you’re keeping and archiving on behalf of clients, will also add to the demands on your storage, but now we’re into considering archiving regimes – whether to use tapes, dedicated NAS/SAN, or even simply burn to DVD. While all sorts of fearsomely complex archiving solutions exist, calling tar from Crontab may be all you need to archive your log files before they’re purged by Logrotate. We’re all for a simple life – after all, that’s why we run Linux.

And speaking of complications, these four pages have just rounded up the basics for you. Rsyslog’s filtering options alone would fill another tutorial, but each reader will have different requirements, so we hope that you can use our intro to get started with a remote logging server, and dig deeper to find all the functionality you need for your own setup. Θ

Wireshark is a very popular and extremely capable network protocol analyser that was developed by Gerald Combs. Wireshark was born in June 2006

when Combs renamed the network tool Ethereal, which he also created, as he was changing jobs and couldn't use the old name anymore. Nowadays, most people use Wireshark

The TCP packet and the IP packet format.

and Ethereal has been consigned to history. Your Linux distribution will have a ready to install package for analyser too, so go ahead and install it.

You may ask what makes Wireshark different to other network analysers – apart from the fact that it's free – and why we're not simply advocating using tcpdump for packet capturing? The main advantage of Wireshark is that it's a graphical application. Capturing and inspecting network traffic using a graphical user interface is a very helpful thing because it cuts through the complexity of network data.

To help the beginner understand Wireshark they will need to understand network traffic. The aim of this article then is to supply a comprehensive introduction to TCP/IP to enable you to come to useful conclusions about the network traffic data you're analysing.

If you run Wireshark as a normal user, you won't be able to use any network interfaces for capturing, because of the default Unix file permission that network interfaces have.

It's more convenient to run Wireshark as root (sudo wireshark) when capturing data and as a normal user when analysing network data. Alternatively, you can capture network data using the tcpdump command line utility as root and analyse it using Wireshark afterwards. Please keep in mind that on a truly busy network, capturing using Wireshark might slow down a machine or, even worse, might not enable you to capture everything because Wireshark needs more system resources than a command line program. In such cases using tcpdump for capturing network traffic is the wisest solution.

Capturing network data

The easiest way to start capturing network packets is by selecting your preferred interface after starting Wireshark and then pressing Start. Wireshark will show network data on your screen depending on the traffic of your network. Note that you can select more than one interface. If you know nothing about TCP, IP or the other TCP/IP protocols, you may find the output complicated and difficult to read or understand. In order to stop the capturing process you just select Capture > Stop from the menu. Alternatively, you can press the fourth icon from the left, the one with a red square (which is shorthand for 'Stop the running live capture') on the Main toolbar (Note: its exact location depends on your Wireshark version). This button can only be pressed while you are capturing network data.

When using the described method for capturing, you can't