Fluentbit does not follow the logs

Sergey.Lysenko — Mon, 14 Feb 2022 14:47:16 +0000

Fluentbit can follow and parse logs and send them to different systems, for example, Newrelic or Elasticsearch. But you may notice, that for some reason it does not follow files or stops follow ones.

By default, the precompiled version of Fluentbit goes with inodes support. And if your logs are on the NFS storage, then you may see warnings:

[2021/12/16 06:29:19] [debug] [input:tail:tail.4] scan_blog add(): dismissed: /var/log/nginx/access.log, inode 2621611

That exactly points to something wrong with inodes. There is a known issue at Fluentbit, where you can find out the details.

To resolve the issue on NFS, you should recompile the Fluentbit without inotify support (FLB_INOTIFY parameter):

cmake -DFLB_INOTIFY=Off

I am using td-agent-bit, and here is my Dockerfile:

ARG FLB_VERSION=1.8.11

FROM debian:bullseye-slim as builder

ARG FLB_VERSION
ARG FLB_TARBALL=https://github.com/fluent/fluent-bit/archive/v$FLB_VERSION.tar.gz
ENV FLB_SOURCE $FLB_TARBALL
RUN mkdir -p /fluent-bit/bin /fluent-bit/etc /fluent-bit/log /tmp/fluent-bit-master/

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    build-essential \
    ca-certificates \
    curl \
    cmake \
    make \
    tar \
    libssl-dev \
    libsasl2-dev \
    pkg-config \
    libsystemd-dev \
    zlib1g-dev \
    libpq-dev \
    postgresql-server-dev-all \
    flex \
    bison \
    && curl -L -o "/tmp/fluent-bit.tar.gz" ${FLB_SOURCE} \
    && cd tmp/ && mkdir fluent-bit \
    && tar zxfv fluent-bit.tar.gz -C ./fluent-bit --strip-components=1 \
    && cd fluent-bit/build/ \
    && rm -rf /tmp/fluent-bit/build/*

WORKDIR /tmp/fluent-bit/build/
RUN cmake -DFLB_RELEASE=On \
          -DFLB_TD=On \
          -DFLB_INOTIFY=Off \
          -DFLB_TRACE=On \
          -DFLB_JEMALLOC=On \
          -DFLB_TLS=On \
          -DFLB_SHARED_LIB=On \
          -DFLB_EXAMPLES=Off \
          -DFLB_HTTP_SERVER=On \
          -DFLB_IN_SYSTEMD=On \
          -DFLB_OUT_KAFKA=On \
          -DFLB_OUT_PGSQL=On ..

RUN make -j $(getconf _NPROCESSORS_ONLN)
RUN install bin/td-agent-bit /fluent-bit/bin/

FROM newrelic/newrelic-fluentbit-output:1.12.1 as newrelic-fluentbit-plugin

FROM debian:bullseye-slim

MAINTAINER Sergey Lysenko

ARG FLB_VERSION

# Update certificates, install ed
RUN apt-get update && \
    apt-get install -y ca-certificates ed && \
    rm -rf /var/lib/apt/lists/*

# Add key for fluentbit
RUN apt-get update && \
    apt-get install -y gnupg && \
    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4FF8368B6EA0722A && \
    apt-get autoremove -y gnupg && \
    rm -rf /var/lib/apt/lists/*

# Install fluentbit
RUN echo "deb https://packages.fluentbit.io/debian/buster buster main" | tee /etc/apt/sources.list.d/fluentbit.list && \
    apt-get update && \
    apt-get install -y td-agent-bit=${FLB_VERSION} && \
    rm /opt/td-agent-bit/bin/td-agent-bit && \
    rm -rf /var/lib/apt/lists/*

# Add fluentbit user
RUN addgroup --system fluentbit && \
    useradd --no-log-init --create-home --home-dir /home/fluentbit --shell /bin/bash --uid 1234567 --system --gid root fluentbit && \
    usermod -a -G fluentbit fluentbit

COPY --from=builder /fluent-bit/bin/td-agent-bit /opt/td-agent-bit/bin/td-agent-bit
COPY --from=newrelic-fluentbit-plugin /fluent-bit/bin/out_newrelic.so /opt/td-agent-bit/bin/plugins/out_newrelic.so

RUN usermod -u 1234567 fluentbit

# Permissions
RUN dirs="/var/log/fluentbit /etc/td-agent-bit" \
    && mkdir -p $dirs \
    && chown -R fluentbit $dirs \
    && chgrp -R 0 $dirs \
    && chmod -R g=u $dirs \
    && chmod 664 /etc/passwd /etc/group

COPY config /etc/td-agent-bit
COPY entrypoint.sh /

USER fluentbit

ENTRYPOINT ["/entrypoint.sh"]

CMD ["/opt/td-agent-bit/bin/td-agent-bit", "-c", "/etc/td-agent-bit/td-agent-bit.conf"]

The post Fluentbit does not follow the logs appeared first on Sergey Lysenko.

How to check memory limit inside Docker container

Sergey.Lysenko — Tue, 11 Jan 2022 15:05:46 +0000

You should know the memory limit of your container (not the whole host) when packing applications there. For example and simplicity, let’s imagine you are creating customized redis, memcached or varnish image and you wish to know available memory when the container starts to set it as the memory limit for your application.

There are two options for it, you can choose one depending on cgroup version (v1 or v2), or check both.

/sys/fs/cgroup/memory/memory.limit_in_bytes
/sys/fs/cgroup/memory.max

So, here is the bash code snippet:

if [ -f "/sys/fs/cgroup/memory/memory.limit_in_bytes" ]; then
    container_memory_limit=$( cat /sys/fs/cgroup/memory/memory.limit_in_bytes )
elif [ -f "/sys/fs/cgroup/memory.max" ]; then
    container_memory_limit=$( cat /sys/fs/cgroup/memory.max )
fi

There in variable container_memory_limit will be a memory limit in bytes. Just in case you can also cast the result value to an integer using AWK (i like it):

container_memory_limit=$( awk -v memory_limit=$container_memory_limit 'BEGIN { printf "%0.0f\n", memory_limit }' )

The post How to check memory limit inside Docker container appeared first on Sergey Lysenko.

Docker Archives - Sergey Lysenko

Fluentbit does not follow the logs

How to check memory limit inside Docker container