Notes: LinuxFest Northwest 2016

April 2016 · 7 minute read

From now on I’m going to post the notes I take at any conferences. For all of our benefit. I’ve annotated and added links as the result of my own research based on these notes. Any of my thoughts will be signed with -CSB.

LFNW 2016 was great. There are always so many good talks at LFNW. I’ve been to a lot of conferences where there are only one or two standout sessions per hour, but every session is great and it is trying to figure out which one you really want to attend.

On Crypto and Fearmongering

Jared Friend, American Civil Liberties Union of Washington.

Great talk from an attorney with ACLU of Washington talking about the end of the “golden age” of law enforcement. The current fight against encryption technologies is tied to the police being spoiled with easy, cheap information about subjects of interest. Previously there was lots of leg work to be done to get someone. Encryption means its truly back to work for the cops again.

The Twelve-Factor Container

Casey West, Pivotal.

This talk was great for me because I have been working a series of Dev Ops articles that focus on taking an app through all twelve factors. Nice to see non technology specific strict implementations. -CSB

This talk is more about operational maturity then containers.

My favorite two quotes from the talk:

Reference: The Twelve-Factor App by Heroku.

  1. One codebase tracked in revision control, many deploys.

    • Deliver same codebase to many environments without modification.
    • Immutable containers are used. Image delivered for multiple environments.
    • Only tag code with semantic versions, not environments.
    • Anti-patterns
      • Building separate images for staging and production.
      • Tags for dev and prod.
    • Best Practices
      • Use environment and / or feature flags.
      • Make environment decisions in the app based on environment variables.

  2. Explicitly declare and isolate dependencies.

    • Building containers allows you to truly isolate dependencies.
    • Anti-pattern: Always using latest version of a dependency. Not using pinned version.
    • Best Practices
      • Declare version numbers of upstream dependencies.
      • Depend on base images for default filesystm and runtimes.
    • Author prefers Alpine Linux as the baseline to build his images.

  3. Store config in the environment.

    • Breaks immutability.
    • Volume mounting configuration files limits portability.
    • Anti-patterns
      • config.yml
      • properties.xml
      • Hard-coded feature flags
    • Best Practice: Literally environment variables.

  4. Treat backing services as attached resources.

    • Should connect to resources as network attachments if available.
    • Imagine there is no local disk. Everything should be accessed as it is available only as a network service.
    • Anti-pattern: Reliance on local disk resources.
    • Best Practice: Connect to network-attached services using connection info from the environment

  5. Strictly separate build and run stages.

    • Output is deployable container from any test. e.g. Docker build -> Docker Run
    • Everything is installed on deployment timeline.
    • Anti-pattern: Install on deploy
    • Best Practices
      • Build immutable images, then run those images.
      • Respect the lifecycle: build, run, destroy as discrete steps.

  6. Execute the app as one or more stateless processes

    • Keep application states externalized.
    • Processes may be run in a distributed environment
    • Anti-pattern example: NFS usage across distributed environments - breaks down at scale.
    • Best Practice: Schedule long running processes by distributing them across a cluster of physical hardware.

  7. Export services via port binding

    • All your services should be exposed as network services via a port.

  8. Scale out via the process model

    • Best Practice: Horizontally scale by adding instances

  9. Maximuze robustness with fast startup and graceful shutdown

    • Fast startup is a natural benefit of containers.
    • Recommended way to quickly make lots of data available is with Redis.

  10. Keep development, staging, and production as similar as possible.

    • Best Practice: Run containers in development.

  11. Treat logs as event streams

    • Use a simple logging mechanism. e.g. stdout / stderr
    • stdout / stderr mechanisms can be gathered from containers to read and stream natively.
    • Best Practice: stdout / stderr
    • Anti-pattern example: Random log files all over the file system
    • Specific tool / app recommendations
    • Kubernete, Mesos, etc. have log aggregation tools as container orchestration services.

  12. Run admin / management tasks as one-off processes.

    • Default commands can exist when spinning up containers. Startup tasks for example.
    • Anti-pattern: Custom container for one off tasks
    • Best Practice: Reuse application images with specific entrypoints for tasks

Overall Notes:

Two days before this conference I found a video that illustrates every anti-pattern Casey talked about. -CSB

It’s Dangerous to go alone, exploring /proc with friends

Alex Juarez, Rackspace. Slides.

# List of open files for a given process.
$ lsof -ngvim
COMMAND   PID  USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
gvim    13681 user  rtd       DIR              252,0     4096         2 /
gvim    13681 user  txt       REG              252,0  2923968   3155726 /usr/bin/vim.gtk
gvim    13681 user  mem       REG              252,0   118432   4327576 /usr/lib/x86_64-linux-gnu/gio/modules/libgioremote-volume-monitor.so
gvim    13681 user  mem       REG              252,0    47600   2622020 /lib/x86_64-linux-gnu/libnss_files-2.23.so
gvim    13681 user  mem       REG              252,0    47648   2622017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
gvim    13681 user  mem       REG              252,0    93128   2622015 /lib/x86_64-linux-gnu/libnsl-2.23.so
gvim    13681 user  DEL       REG                0,5          393314314 /SYSV00000000
gvim    13681 user  mem       REG              252,0   704128   3156216 /usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf


# All file knowledge
$ stat /usr/bin/vim.gtk
  File: '/usr/bin/vim.gtk'
  Size: 2923968   	Blocks: 5712       IO Block: 4096   regular file
Device: fc00h/64512d	Inode: 3155726     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-04-27 16:11:12.751952740 -0700
Modify: 2016-04-08 05:44:36.000000000 -0700
Change: 2016-04-25 15:43:01.135585285 -0700
 Birth: -

# What was passed to the kernel at boot time
$ /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.19.0-58-generic root=/dev/mapper/computer--vg-root ro

# Unlike many other things, PROC file output can be sparsely documented and widely change between versions.
man proc

# file contains the command line for the process
$ cat /proc/<pid>/cmdline

# All environment variables a PID has run with
$ cat /proc/<pid>/environ

# lists all files used by a current process
$ ls -la /prof/pid/fd

# Score = OS score. Adjust = Can set something as more or less important to kill.
$ cat /proc/pid/oom_adj
$ cat /proc/pid/oom_score

# Current process commandline
# Can be used in a script in case of dynamic environment creation.
$ cat /proc/self/cmdline

# See what files a command opens
$ strace -e trace=open lspci

Redis functions and structures

Dave Nielsen, Redis Labs.

Fav quote: “You know, Cockroaches, startups focused on revenue and customers instead of high valuations that can survive the upcoming nuclear valuation.” - Dave Nielsen

Docker + Ansible = The Best of both worlds

Mike Titus, Conversica.

“We’ve moved to infrastructure as code.” - Mike Titus

Let’s Encrypt, Our First Half-million certs

Seth Schoen, Electronic Frontier Foundation.

I had to see these guys. The more interesting parts of the talk were about how they went about becoming a Certificate Authority and all the bullshit they went through to achieve this process. The only curious part is the non-disclosure of certain relationships. -CSB

Another fantastic conference! See you friends, new and old, next year!