1. OVERVIEW

   1. To implement a set of libraries for L4 protocol processing (UDP, TCP etc.) for both IPv4 and IPv6.

   The goal is to provide lightweight, high performance and highly adaptable implementation for L4(UDP, TCP etc.) protocol processing. The provided API are not planned to be compatible with BSD socket API. These libraries are supposed to be built on top of DPDK.

   Note: these libraries are not supposed to be a ‘complete’ network stack.

   Implementation of ARP, IP, ETHER, etc. layers and related routing tables, code for setup, manage and perform actual IO over underlying devices are all out of scope of these libraries.

   The only information these libraries need about the underlying devices:

   • supported HW offloads
   • MTU and L3/L2 addresses That allows the libraries to fill L2/L3 headers and mbuf metadata for the outgoing packets.

   These libraries should be developed in such manner, that they could be used independently from implementations of 2) and 3).

   2. To create VPP graph nodes, plugins etc using those libraries to implement a host stack.

   3. To create such mechanisms (netlink agents, packaging, etc) necessary to make the resulting host stack easily usable by existing non-vpp aware software.

   The library uses siphash logic from the below source https://github.com/veorq/SipHash

2. APIs

   TLDK provides three series of APIs:

   • TLDK native APIs, provided by libtle_l4p.
   • Posix APIs, provided by libtle_glue with PRELOAD compile macro.
   • Posix APIs with changed symbol names, provided by libtle_glue without PRELOAD macro.

3. INSTALLATION GUIDE

   • Original guide

   ---

   1. Obtain latest supported DPDK version and build it. (refer to http://dpdk.org for information how to download and build it). Currently supported(tested) DPDK versions: 16.11 LTS, 17.11 LTS, 18.02.
   2. Make sure that RTE_SDK and RTE_TARGET DPDK related environment variables are setup correctly.
   3. Go to the TLDK root directory and type: ‘make all’.
   4. Run sample applications.

   For enabling unit tests application using GoogleTest please see: ./test/gtest/README

   As an example: export RTE_SDK=/opt/DPDK export RTE_TARGET=x86_64-native-linuxapp-gcc

   cd tldk make all ./x86_64-native-linuxapp-gcc/app/l4fwd …

• For preload use

---

Debug:

 $ make DPDK_DEBUG=y EXTRA_CFLAGS="-g -O0 -fPIC -DPRELOAD" all

Release:

 $ make EXTRA_CFLAGS="-g -fPIC -DPRELOAD" all

• For TLDK API use

---

Debug:

 $ make DPDK_DEBUG=y EXTRA_CFLAGS="-g -O0 -fPIC" all

Release:

 $ make EXTRA_CFLAGS="-g -O3 -fPIC" all

3. CONTENTS

   $(TLDK_ROOT) | +—-app | | | +– nginx - a clone of nginx integrated with TLDK | (refer to app/nginx/README.TLDK for more information) | +—-lib | | | +–libtle_dring - dring library | | | +–libtle_l4p - implementation of the TCP/UDP packet processing | | | +–libtle_timer - implementation of the timer library | | | +–libtle_glue - socket glue layer with arp, icmp, epoll, etc | +—-examples | | | +–l4fwd - sample app to demonstrate and test libtle_l4p TCP/UDP | usage (refer to examples/l4fwd/README for more information) | +—-test - unit-tests | | | +–dring - UT for libtle_dring (standalone app) | | | +–gtest - UT for libtle_dring, libtle_l4p and libtle_timer | | (googletest) | | | +–timer - UT for libtle_timer (standalone app) | | | +–packetdrill - UT for stack (standalone app)

5. Features

   Done:

   • posix interface
   • loopback device
   • regression test
   • multi-thread
   • lightweight mem
   • tcp_info (paritial)
   • fd management
   • arp request/reply
   • icmp reply
   • interrupt mode
   • blocking recv/send
   • TSO
   • UFO

   TODO:

   • fuzzing
   • SACK
   • RACK
   • zerocopy APIs
   • batching APIs
   • multi-process
   • numa awareness
   • context recycle on thread exit

6. Thread model

   • Multi-process is still not fully supported.

   • Symmetric multi-thread

     (app thread) (app thread) (app thread)

       \                    \                     \
       /                    /                     /
       \                    \                     \

     ---

     FD management, Socket APIs (FE)

     ---

     | | | | | | | ctx | | ctx | | ctx | | | | | | |

     ---

        \__                  |                   __/
           \__               |                __/
              \__            |             __/
                 \__         |          __/
                   -------------------------
                   |    (RSS) NIC (FDIR)   |
                   -------------------------

   • Lookaside multi-thread

     (app thread) (app thread) (io thread)

       \                     \                    \
       /                     /                    /
       \                     \                    \

     ---

     FD management, Socket APIs (FE)
     /

                                                  \
                                                  /

     ---

     | | | ctx |

                            |
                -------------------------
                |          NIC          |
                -------------------------

7. How to run

   We have two setups which need their own preparation.

   • virtio-user: test with virtio-user + vhost-kernel;
   • physical NIC: test with physical NIC bound to vfio.

   If you are using physical NIC:

   
   a. Set Linux boot options (Only needed if you will use physical NIC)
     a1. Add below configuration into GRUB_CMDLINE_LINUX in /etc/default/grub
       "intel_iommu=on iommu=pt"
   
     a2. Update grub
       $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
   
     If you want to use 1GB hugepages, you can also add below content in the
     boot cmdline:
       "default_hugepagesz=1G hugepagesz=1G hugepages=2"
   
   b. Adjust RLIMIT_MEMLOCK (Only needed if you will use physical NIC)
     Add below two lines into /etc/security/limits.conf
       "* soft memlock 4194304
        * hard memlock 4194304"
   
   c. Reboot system
   
   d. Bind NIC to vfio-pci
   
     $ sudo modprobe vfio-pci
     $ sudo ./usertools/dpdk-devbind.py -b vfio-pci 0000:01:00.1
     $ sudo chmod 666 /dev/vfio/16 (16 is just an example)
   
   If you are using virtio-user:

   a. Prepare vhost-kernel

   $sudomodprobevhost-net(ifyoudo{n}^{\mathrm{\prime }}thavethosemodules,havetocompilebyyourself)sudo\; modprobe\; vhost-net\; (if\; you\; don\text{'}t\; have\; those\; modules,\; have\; to\; compile\; by\; yourself)$ sudo chmod 666 /dev/vhost-net $ sudo tunctl -u

   b. Prepare the vNIC

   $ export DPDK_VNIC=”–vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024,iface=tap0”

   For both cases, we need to: ~~~~~~~~~~~~~~~~~~~~~~~~~~

   $sudochmod777\mathrm{/}dev\mathrm{/}hugepagessudo\; chmod\; 777\; /dev/hugepages$ export DPDK_IP=1.1.1.1

   Note: for specific test example, you can refer to the example commit log.

8. How to run packetdrill tests:

Compile it in LOOK_ASIDE_BACKEND mode:

$ make PACKETDRILL=y EXTRA_CFLAGS="-g -O0 -fPIC -march=native -DLOOK_ASIDE_BACKEND " all

To run it:

$ test/packetdrill/packetdrill --local_ip=192.168.0.2 \
    --remote_ip=192.0.2.1 --so_filename=`pwd`/libtldk.so \
    test/packetdrill/tests/tldk/fast_retransmit/fr-4pkt-tldk.pkt

8. Tested Examples

   • examples/client
   • examples/server
   • wget (epoll)
   • curl (poll)
   • haproxy (multi-thread mode)