Docker
Fluentd
ログ収集を行うためにfluentdを使用するが、その前提となる環境構築メモ。
公式を参考に進める。
.
├── docker-compose.yml
└── fluentd
└── etc
└── fluent.conf
# docker-compose.yml
version: '3'
services:
fluentd:
image: fluent/fluentd:v1.11-1
volumes:
- ./fluentd/etc:/fluentd/etc
fluent.confは、とりあえずHello worldをやってみるため、こちらを参考に設定する。
# fluent.conf
<source>
@type dummy
dummy {"hello":"world"}
</source>
Dockerを起動させてみる。
docker-compose up
fluentd_1 | 2020-07-24 13:02:19 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd_1 | 2020-07-24 13:02:19 +0000 [info]: gem 'fluentd' version '1.11.1'
fluentd_1 | 2020-07-24 13:02:19 +0000 [error]: config error in:
fluentd_1 | <source>
fluentd_1 | @type dummy
fluentd_1 | dummy {"hello":"world"}
fluentd_1 | </source>
fluentd_1 |
fluentd_1 | 2020-07-24 13:02:19 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'tag' parameter is required"
fluentd_fluentd_1 exited with code 1
エラーになった…
エラーメッセージを見るとtagが必須とのこと。
fluent.confを次のように修正した。
<source>
@type dummy
dummy {"hello":"world"}
tag hello_world
</source>
修正後、Dockerを起動する。
docker-compose up
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: gem 'fluentd' version '1.11.1'
fluentd_1 | 2020-07-24 13:04:10 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:04:10 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: using configuration file: <ROOT>
fluentd_1 | <source>
fluentd_1 | @type dummy
fluentd_1 | dummy {"hello":"world"}
fluentd_1 | tag "hello_world"
fluentd_1 | </source>
fluentd_1 | </ROOT>
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: adding source type="dummy"
fluentd_1 | 2020-07-24 13:04:10 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:04:10 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
fluentd_1 | 2020-07-24 13:04:10 +0000 [info]: #0 fluentd worker is now running worker=0
fluentd_1 | 2020-07-24 13:04:11 +0000 [warn]: #0 no patterns matched tag="hello_world"
fluentd_1 | 2020-07-24 13:04:12 +0000 [warn]: #0 no patterns matched tag="hello_world"
fluentd_1 | 2020-07-24 13:04:14 +0000 [warn]: #0 no patterns matched tag="hello_world"
fluentd_1 | 2020-07-24 13:04:18 +0000 [warn]: #0 no patterns matched tag="hello_world"
エラーはなくなったが[warn]: #0 no patterns matched tag="hello_world"と表示される。
これは、インプットに対してのアウトプットが存在しないことが原因の模様。
そこで、fluent.confを次のように修正する。
<source>
@type dummy
dummy {"hello":"world"}
tag hello_world
</source>
<match hello_world.**>
@type stdout
</match>
修正後、Dockerを起動する。
docker-compose up
fluentd_1 | 2020-07-24 13:07:38 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: gem 'fluentd' version '1.11.1'
fluentd_1 | 2020-07-24 13:07:39 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:07:39 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: using configuration file: <ROOT>
fluentd_1 | <source>
fluentd_1 | @type dummy
fluentd_1 | dummy {"hello":"world"}
fluentd_1 | tag "hello_world"
fluentd_1 | </source>
fluentd_1 | <match hello_world.**>
fluentd_1 | type stdout
fluentd_1 | </match>
fluentd_1 | </ROOT>
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
fluentd_1 | 2020-07-24 13:07:39 +0000 [warn]: #0 'type' is deprecated parameter name. use '@type' instead.
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: adding match pattern="hello_world.**" type="stdout"
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: adding source type="dummy"
fluentd_1 | 2020-07-24 13:07:39 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:07:39 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
fluentd_1 | 2020-07-24 13:07:39 +0000 [info]: #0 fluentd worker is now running worker=0
fluentd_1 | 2020-07-24 13:07:40.084009100 +0000 hello_world: {"hello":"world"}
fluentd_1 | 2020-07-24 13:07:41.001777200 +0000 hello_world: {"hello":"world"}
fluentd_1 | 2020-07-24 13:07:42.021463500 +0000 hello_world: {"hello":"world"}
fluentd_1 | 2020-07-24 13:07:43.037985700 +0000 hello_world: {"hello":"world"}
期待した内容が出力されるようになった。
fluentdがなんとなく動くことが確認できた。
最終的にはログをElasticsearchやS3に置いていきたいので引き続き調査を進める。