Context

In our Kubernetes cluster, in order to control connectivity between pods, we're using Istio as a service mesh. Practically, it means that every pod in our cluster has an envoy proxy attached to it, capturing traffic on both way : ingress and egress.

By default, every pod can reach any other pod within the mesh, however we want to be able to only authorize traffic to legitimate applications and forbid everything else.

It may sound simple : after all an Istio resource exists to do that, but we faced some caveats during the deployment. This article deep dive into the egress filtering implementation and demonstrate what is exacly modified in the envoy configuration when you manipulate Istio resources.

The Istio Control Plane

The brain of our service mesh is istiod : aka the Istio control plane.

This piece of software is responsible to :

• gather the configuration described in the Istio resources (Virtual services, ServiceEntry, DestinationRules...) but also in Kubernetes itself (Services, Endpoints…)
• build the corresponding envoy configuration
• push the configuration to every envoy proxy in the mesh.

In this page I usually use the term of “istio proxy” instead of “envoy proxy”, but they are strictly equivalent. Envoy is the real name of the proxy software used by Istio.

Envoy terminology

The following is a simplified view of Envoy.

Excerpt from the official Envoy documentation:

Listener: A listener is a named network location (e.g., port, unix domain socket, etc.) that can be connected to by downstream clients. Envoy exposes one or more listeners that downstream hosts connect to.

Cluster: A cluster is a group of logically similar upstream hosts that Envoy connects to. Envoy discovers the members of a cluster via service discovery.

I used it to draw the images down below, to specify in which logical block the traffic is entering to depending on the direction (ingress or egress)

Simple Istio mesh config

Before jumping into the details of the egress filtering, let's have an overview of the envoy configuration when your pods are deployed with an Istio proxy without any specific configuration.

Here, we have an application-0 which expose a service on 8080 HTTP port and has 3 dependencies : application-1 , application-2 located inside the mesh, and bigtable which is located outside of the mesh.

This is resulting in the proxy config as :

• No specific inbound listener
• More than 2 outbound clusters : all services in the mesh are added in term of outbound configuration

It means that every topology change updates the configuration to all proxy in the mesh, even if your application is not concerned. Envoy will then consume more CPU and memory to process this complete mesh topology.

Adding mesh internal dependencies

To avoid this overload and most importantly regarding this article subject, to qualify our workload dependencies, we usually leverage the Sidecar resource which allows to fine tune ingress and egress traffic of our workload.

Let's start with only our mesh internal dependencies : application-1 and application-2

To do that we define in the Sidecar resource :

• 1 ingress
• 2 egress

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: "release-name"
spec:
  workloadSelector:
    labels:
      app: "release-name"
  outboundTrafficPolicy:
    mode: "ALLOW_ANY"
  ingress:
  - port:
      number:  8080
      protocol: TCP
      name: http
    defaultEndpoint: "127.0.0.1:8080"
  egress:
  - hosts:
    - namespace/application-1
    - namespace/application-2

This is resulting in the proxy config as :

• 1 inbound listener : this is mainly used to have correct metrics. However at this point if you mistake the port number, you gonna break your inbound traffic.
• 2 outbound clusters : so all other services in the mesh are ignored in terms of outbound configuration, but they’re still reachable.

The first benefit is a lower footprint : the configuration is now reduced as only a couple of dependencies. We're almost ready to leverage the filtering, but right now we're only using the ALLOW_ANY parameter, so nothing is really filtered out. Yes, but it may have an impact anyway.

Harmless, really ?

Indeed, the application-3 is not a dependency declared in the Sidecar resource, so the internal outboundCluster application-3 doesn't exist anymore.

If application-0 must reach application-3 (forgotten dependency for example) like I said before, the traffic is allowed, thanks to  outBoundTrafficPolicy  set to ALLOW_ANY.

However, it may break if application-3 have specific cluster configuration (VirtualService, DestinationRules...). Every traffic forwarded througth the passThroughCluster, is done without taking destination VS and DR configuration into consideration. So take extra precautions to really identify your dependencies before adding the Sidecar resource.

Regarding external services like bigtable , nothing changes : all traffic forwarded is still using the passThroughCluster . We must deal with this before actually performing the filtering.

Adding mesh external dependencies

Now we've defined all internal dependencies, let's add the external ones.  Open your Sidecar resource in your favorite editor and add bigtable .

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: "release-name"
spec:
  workloadSelector:
    labels:
      app: "release-name"
  outboundTrafficPolicy:
    mode: "ALLOW_ANY"
  ingress:
  - port:
      number:  8080
      protocol: TCP
      name: http
    defaultEndpoint: "127.0.0.1:8080"
  egress:
  - hosts:
    - namespace/application-1
    - namespace/application-2
    - ./bigtable.googleapis.com

and you create a ServiceEntry resource which specify :

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: "release-name"
  namespace: istio-system
spec:
  exportTo:
  - '.'
  hosts:
  - bigtable.googleapis.com
  location: MESH_EXTERNAL
  ports:
  - name: https
    number: 443
    protocol: HTTPS
  resolution: DNS

This is resulting in the proxy config as :

3 outbound clusters. The 3rd one (corresponding to bigtable.googleapis.com) finally shows up as soon as you create the Service Entry resource.

Why not before ? Because Istio rely on it's service registry to build its cluster configuration. All Kubernetes services are by default populated in the Istio registry, but bigtable is not part of the mesh : that's why we create a Service Entry resource which creates an entry in the Istio Service Registry.

Now you can activate the REGISTRY_ONLY option, which only allow traffic from every logical cluster and forbid anything which used to go through the passThroughCluster.

I frequently use it to wait something like 10 sec for apt to finish its unattended upgrades and release the apt lock, like this :

(
flock -w 10 9 || exit 1
aptly repo remove $repo $files
) 9>/var/lock/aptlock

rsync -avz -e 'ssh -p 2223' <origin> <destination>

By default, it keeps 20 versions but it can be overridden. It manage APTLY package query : so it could be run against a single package or a set of packages.

Simple package query example

# clean-repo.py --repo buster-dev --package-query vault-server --keep 2 --dry-run
Run in dry mode, without actually deleting the packages.
Remove "vault-server" from buster-dev and keep the last 2 packages.

This package(s) would be kept:
vault-server_1:0.11.4~20190424~buster.build0_amd64
vault-server_1:0.11.4~20190425~buster.build0_amd64

# clean-repo.py --repo buster-dev --package-query vault-server --keep 1 --dry-run
Run in dry mode, without actually deleting the packages.
Remove "vault-server" from buster-dev and keep the last 1 packages.

This package(s) would be kept:
vault-server_1:0.11.4~20190425~buster.build0_amd64

This package(s) would be deleted:
vault-server_1:0.11.4~20190424~buster.build0_amd64

Multiple package query example

# clean-repo.py --repo buster-dev --package-query 'Name (% vault-*)' --keep 1 --dry-run
Run in dry mode, without actually deleting the packages.
Remove "Name (% vault-*)" from buster-dev and keep the last 1 packages.

This package(s) would be kept:
vault-common_1:0.11.4~20190425~buster.build0_amd64
vault-server_1:0.11.4~20190425~buster.build0_amd64

This package(s) would be deleted:
vault-common_1:0.11.4~20190424~buster.build0_amd64
vault-server_1:0.11.4~20190424~buster.build0_amd64

Full script

Use your favorite scheduler (rundeck, cron...) to run this script.

sudo apt-get install libpam-google-authenticator

Configure google-authenticator

Run google-authenticator as the user you want to be 2FA's authenticated and answer a few questions.

Shall this tool update your configuration file ? Answer yes to this first question.

For max security :

• Restrict the use of a token by waiting between login
• Token Time Window : 30 sec.
• Attempts numbers : 3

The last one helps to prevent brute-force login attacks.

Scan the QRCode (or enter key code) with your favorite smartphone.  

Make your SSH config

Open /etc/ssh/sshd_config and make sure to have this gobal configuration settings :

UsePAM yes
ChallengeResponseAuthentication yes

While you can use the following parameters gobally, I personnaly prefer having a per-user config :

Match User fred
     AuthenticationMethods publickey,password-interactive

Make your PAM config

Open /etc/pam.d/sshd with your favorite editor :

sudo vim /etc/pam.d/sshd

Add before "@include common-auth" section the following line

auth required  pam_google_authenticator.so

Restart your SSH server.

Test

For your own safety, keep your current SSH session opened and from another window open a new SSH session to test your new 2FA authentication.

Alternative

If you want to get rid of your password prompt and just rely on your SSH key + your OTP then adjust your SSH pam config like this :

# Standard Un*x authentication.
#@include common-auth
auth [success=1 default=ignore]   pam_google_authenticator.so
auth    requisite           pam_deny.so
auth    required            pam_permit.so

Comment the default common-auth entry which is the section which actually prompt for a password and report the two new 'auth' lines.

Ces TP utilisent une VM LinuxMint (what else ?!) dont vous pouvez télécharger l'ISO customisé ici.

TP Réseaux, MMI, première année, semestre 1

TP1 : calculs d'espaces d'adressage IP

TP Réseau 1

TP réseau 1 corrigé

TP 2 : Prise en main de Wireshark, premières captures et interprétations

TP Réseau 2

TP réseau 2 corrigé

TP 3 : Ce TP a pour objectif de mettre en pratique la capture de trame ethernet, et de mettre en évidence la fragmentation de datagrammes.

TP Réseau 3

TP Réseau 3 corrigé

TP Système d'exploitation, MMI, première année, semestre 1

TP1 : Ce TP est destiné à vous familiariser avec la notion de virtualisation, et la manipulation d'un système d'exploitation.

TP Système d'exploitation 1

TP2 : Ce TP a pour objectif la prise en main d’une distribution GNU/Linux, Linux Mint 17. L'interface graphique est relativement similaire à ce qui existe sous Windows ou MacOSX. Nous allons donc nous intéresser aux spécificités de cet OS. Et pour commencer : le shell.

TP Système d'exploitation 2

TP Système d'exploitation 2 corrigé

TP3 : Ce TP vous propose de rédiger quelques scripts en bash.

TP Système d'exploitation 3

TP Système d'exploitation 3 corrigé

TP4 : Ce TP aborde la gestion des comptes utilisateurs sous linux, introduit la notion de privilège et par conséquent la notion de sécurité. Ce TP fait beaucoup appel aux commandes vues précédemment.

TP Système d'exploitation 4

TP5 : Ce TP aborde les outils nécessaires afin de gérer les logiciels installés sur Linux.

TP Système d'exploitation 5

TP Services sur réseaux, MMI, première année, semestre 2

TP1 : Découverte de Packet Tracer : outil de simulation de réseaux.

TP Service Sur Réseau 1

TP Service Sur Réseau 1 corrigé

TP2 : Ce TP a pour objectif de mettre en application les VLAN et d’introduire la notion de sécurité d’accès au réseau physique par les mécanismes de limitation de propagation de VLAN et de sécurité lié au port.

TP Service Sur Réseau 2

TP Service Sur Réseau 2 corrigé

TP3 : Ce TP propose d’étudier le protocole spanning tree. Sa bonne compréhension est indispensable afin d’assurer la stabilité d’un réseau local.

TP Service Sur Réseau 3

TP4 : Ce TP propose la mise en oeuvre de routage inter-VLAN, de routage statique et présente le principe de l’agrégation de routes.

TP Service Sur Réseau 4

TP Service Sur Réseau 4 corrigé

TP5 : Ce TP propose la mise en oeuvre de filtrage. Le réseau que nous allons étudier est constitué d’une zone privée, que nous pouvons comparer au réseau d’une petite entreprise. Un ISP fourni un routeur, sur lequel sont branchés les équipements du client.

TP Service Sur Réseau 5

TP Service Sur Réseau 5 corrigé

TP Services sur réseaux, MMI, seconde année, semestre 1

TP1 : Ce TP a pour objectif la prise en main du shell, l'interpréteur de commande des distributions GNU/Linux. La maîtrise de cet outil est indispensable pour bien aborder ensuite l'installation et la configuration des composants de type serveur web, serveur de messagerie, etc.

TP Service Sur Réseau 1

TP Service Sur Réseau 1 corrigé

TP2 : Ce TP aborde la gestion des comptes utilisateurs sous linux, introduit la notion de privilège et par conséquent la notion de sécurité. Ce TP fait beaucoup appel aux commandes vues précédemment.

TP Service Sur Réseau 2

TP Service Sur Réseau 2 corrigé

TP3 : Ce TP permet d'installer un DNS (bind) sur un système Linux, à savoir Linux Mint.

TP Service Sur Réseau 3

TP Service Sur Réseau 3 corrigé

L'objectif des TP suivants est d'installer un serveur web LAMP (Linux / Apache / MySQL / PHP ).

TP5 : Installation d'Apache

TP Service Sur Réseau 5

TP Service Sur Réseau 5 corrigé

TP6 : Configuration avancée d'Apache.

TP Service Sur Réseau 6

TP Service Sur Réseau 6 corrigé

TP7 : Finalisation : mysql et PHP

TP Service Sur Réseau 7

TP Service Sur Réseau 7 corrigé

Simple version could be like this :

find . -type d -maxdepth 1 -name "??*" -exec sh -c "cd '{}' && git pull; cd ..  " \;

The idea is simple, it parse every folder down your current one, and execute `git  pull` inside, then continue to the next folder.

I'm lazy ... I certainly don't want to type this long command every time, so I added an alias. Add this line to your .bash_profile (or .bashrc whatever).

This is a little bit more complex version which avoid digging into .git folder and write useful feedback on your terminal screen.

alias gla='find . -type d -maxdepth 4 -not -path "*/.git*" -name "??*"  -exec bash -c "cd \"{}\"; [ -d .git ] && ( echo ------------ Updating {} ; git pull --all) ; cd .. ; " \;'

Reload your profile :

source ~/.bash_profile

Source : http://www.unixfu.ch/how-to-authenticate-sudo-with-touchid

Simple script used to replace journal disk. I regenerate journal UUID for convenience but you can also keep as it is.

The layout is the following :
disk /dev/sda : 5 journal partitions -> 5 OSD (OSD.40 to OSD.44)
disk /dev/sdb : 5 journal partitions -> 5 OSD (OSD.45 to OSD.49)
disk /dev/sdc : 5 journal partitions -> 5 OSD (OSD.50 to OSD.59)

First, set the noout flag to prevent any data migration.

ceph osd set noout

Then stop each OSD which are related to the first journal disk.

#!/bin/bash

partitions="1 2 3 4 5"
num=40
for partition in $partitions
do
systemctl stop ceph-osd@$num.service
ceph-osd -i $num --flush-journal
((num++))
done

Hotswap the first journal disk, then execute the following script :

#!/bin/bash

partitions="1 2 3 4 5"
# First OSD number
num=40
# Disk
disk=sda
for partition in $partitions
do
journal_uuid=$(uuidgen)
sgdisk --new=$partition:0:+5120M --change-name=$partition:'ceph journal' --partition-guid=$partition:$journal_uuid --typecode=$partition:$journal_uuid --mbrtogpt -- /dev/$disk
partprobe /dev/$disk

mv /var/lib/ceph/osd/ceph-$num/journal /var/lib/ceph/osd/ceph-$num/journal.old
ln -s /dev/disk/by-partuuid/$journal_uuid /var/lib/ceph/osd/ceph-$num/journal

mv /var/lib/ceph/osd/ceph-$num/journal_uuid /var/lib/ceph/osd/ceph-$num/journal_uuid.old
echo $journal_uuid > /var/lib/ceph/osd/ceph-$num/journal_uuid

chown -R ceph. /var/lib/ceph/osd/ceph-$num/journal
chown ceph. /var/lib/ceph/osd/ceph-$num/journal_uuid

ceph-osd -i $num --mkjournal
chown ceph. /dev/$disk$partition
systemctl start ceph-osd@$num.service
((num++))
done

Process if necessary with sdb and sdc and adjust the OSD start number in the script.

Last step : unset the noout flag

ceph osd unset noout

Source file :
Github link

Reference:
Sébastien Han blog

Issue

Out-of-the-box, CSF + OpenVPN don't work together. Well, not with further configuration.

Let's say your OpenVPN conf is working, but you just can't get out on the internet through your VPN box.

Seems we need some masquerade and maybe other things..

Configuration

Here is my conf :

eth0 : external nic
10.8.0.0/24 : my tunnel network

Solution

Don't forget to enable packet forwarding :
echo 1 > /proc/sys/net/ipv4/ip_forward

Edit your sysctl file to make your change permanent.

Then create the following file in your csf folder :

vi /etc/csf/csfpost.sh

Enter the following :

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE iptables -A FORWARD -s 10.8.0.0/24 -j ACCEPT iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

Relaunch csf :

csf -r

Enjoy. De nada.

A bonding configuration allow you to increase your bandwidth by merging several physical interfaces to a virtual one. LACP is a well known aggregation protocol, available in most switches and in Linux. This post will show you how to use LACP between a switch and a Linux server.

Linux

Load the kernel module

cat > /etc/modprobe.d/bond.conf << EOF
alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1
EOF

Disable NetworkManager

Then, you need to disable NetworkManager.

systemctl stop NetworkManager.service
systemctl disable NetworkManager.service

Configure your physical interfaces

Choose your 2 (or more) network interfaces you want to join together. Their configuration must be the same : ethernet type (TX versus LX), speed, duplex, etc...

cat > /etc/sysconfig/network-scripts/ifcfg-p1p1 << EOF
BOOTPROTO="none"
NM_CONTROLLED=no
DEVICE="p1p1"
NAME=p1p1
ONBOOT=yes
SLAVE=yes
MASTER=bond0
EOF

Second interface :

cat > /etc/sysconfig/network-scripts/ifcfg-p1p2 << EOF
BOOTPROTO="none"
NM_CONTROLLED=no
DEVICE="p1p2"
NAME=p1p2
ONBOOT=yes
SLAVE=yes
MASTER=bond0
EOF

Create your virtual interface

Now you can add a new interface bond0, which would be your virtual network interface :

cat > /etc/sysconfig/network-scripts/ifcfg-bond0 << EOF
BOOTPROTO="none"
DEVICE="bond0"
NAME=bond0
IPADDR=192.168.0.45
PREFIX=24
DNS1=192.168.0.2
GATEWAY=192.168.0.1
DOMAIN=local
DEFROUTE=yes
ONBOOT=yes
PEERDNS=yes
PEERROUTES=yes
USERCTL=no
EOF

Switch part

Let's now configure the switch. That's pretty simple. Just don't forget to apply any further configuration to the port channel interface, not the underlying physicals ones. If the configuration is different between 2 channel group member, then the port channel will be disabled and a traffic interruption will occur.

Also, configure the port channel using active mode. Which means the switch tries to "negotiate" the channel. Other configuration options are passive (wait for LACP negotiation) or on (unconditionnaly build the port channel).

On Cisco switches you could add up to 8 members to your channel group.

Cisco IOS

interface GigabitEthernet2
description SERVER1_p1p1
channel-group 1 mode active
!
interface GigabitEthernet3
description SERVER1_p1p2
channel-group 1 mode active
!
interface Port-channel1
description SERVER1`
!

Dell

The Dell syntax is the same as Cisco IOS.

interface Te1/0/1
channel-group 1 mode active
description "SERVER1_p1p1"
!
interface Te1/0/2
channel-group 1 mode active
description "SERVER1_p1p2"
!
interface port-channel 1
description "SERVER1"
!

Check your configuration

On your Linux box

cat /proc/net/bonding/bond0

On your switch

Dell

show interfaces status port-channel 1

Cisco

show interface po 1

Recently I wanted to monitor some servers. In the old days I used Nagios or Cacti for such a purpose (both based on rrdtool) but I wanted to change and let Zabbix a chance.

Zabbix turn out a nice product I have to say. When it's easy to tweak, then I tag it :)

So, I wanted to monitor running virtual machines, but the thing is they are not accessible from the Zabbix server. The hypervisor must send all VM statistics to the server.

In the little script I wrote, I just demonstrate the way I get all running VM from a qemu domain, and then create a corresponding host into Zabbix.

At the end, I create an item (CPU util but it could be something else) and create a graph with it.

What I need to do next : a little script to update the cpu time and send the value to the server.

Why do I need ?

First, get Nova instance name from Nova. Since this is not mandatory, it permit you to get VM names. More elegant.

Then get a ceilometer object and perform a request.

Get the server list from Nova

Retrieve environment variables

os_username = os.environ.get('OS_USERNAME')
os_password = os.environ.get('OS_PASSWORD')
os_regionname = os.environ.get('OS_REGION_NAME')
os_project_name = os.environ.get('OS_PROJECT_NAME')
os_auth_url = os.environ.get('OS_AUTH_URL')

Then, create a nova client object :

novaclient = client.Client(os_username,
             os_password, 
             os_project_name, 
             os_auth_url, 
             service_type="compute", 
             region_name=os_regionname)

Get server list :

servers = novaclient.servers.list(detailed=True)

Ceilometer

Get now a ceilometer object :

ceilometerclient = ceilometerclient.client.get_client(2,
         os_username=os_username,
         os_password=os_password,
         os_tenant_name=os_project_name,
         os_auth_url=os_auth_url,
         region_name=os_regionname)

Create the query

Pick a server id from the list you just obtain the step before.

query = [dict(field='resource_id', op='eq', value=server.id)]

Get the results

cpu_util = ceilometerclient.samples.list(meter_name='cpu_util', limit=1, q=query)    

Full script is here :

If csf warn you everyday while trying to being updated with this message :

URLGET set to use LWP but perl module is not installed, reverting to HTTP::Tiny

Then install libwww-perl

sudo apt-get install libwww-perl

or

yum install perl-libwww-perl

under CentOS like.

Sometimes, your want to change your hostname, more precisely you want your hostname actually reflecting your DNS change.

I won't talk when such a situation happen, but it could happen. So here is how I manage it. This above script runs under Ubuntu, but it should also runs under any distro which use NetworkManager.

Drop this file into /etc/NetworkManager/dispatcher.d

Basically, it perform a DNS request, then compare the result to the hostname already defined. If the hostname has changed, the script update the network system files.