Fluentd输出插件:rewrite_tag_filter用法详解
# Configuration
<match app.component>
@type rewrite_tag_filter
<rule>
key message
pattern /^(w+)$/
tag $1.${tag}
</rule>
</match>
+------------------------------------------+ +------------------------------------------------+
| original record | | rewritten tag record |
|------------------------------------------| |------------------------------------------------|
| app.component {"message":"[info]: ..."} | +----> | info.app.component {"message":"[info]: ..."} |
| app.component {"message":"[warn]: ..."} | +----> | warn.app.component {"message":"[warn]: ..."} |
| app.component {"message":"[crit]: ..."} | +----> | crit.app.component {"message":"[crit]: ..."} |
| app.component {"message":"[alert]: ..."} | +----> | alert.app.component {"message":"[alert]: ..."} |
+------------------------------------------+ +------------------------------------------------+
# for td-agent2 (with fluentd v0.12)
$ sudo td-agent-gem install fluent-plugin-rewrite-tag-filter -v 1.6.0
# for td-agent3 (with fluentd v0.14)
$ sudo td-agent-gem install fluent-plugin-rewrite-tag-filter
<source>
@type tail
path /var/log/httpd/access_log
<parse>
@type apache2
</parse>
tag td.apache.access
pos_file /var/log/td-agent/apache_access.pos
</source>
<match td.apache.access>
@type rewrite_tag_filter
capitalize_regex_backreference yes
<rule>
key path
pattern /\.(gif|jpe?g|png|pdf|zip)$/
tag clear
</rule>
<rule>
key status
pattern /^200$/
tag clear
invert true
</rule>
<rule>
key domain
pattern /^.+\.com$/
tag clear
invert true
</rule>
<rule>
key domain
pattern /^maps\.example\.com$/
tag site.ExampleMaps
</rule>
<rule>
key domain
pattern /^news\.example\.com$/
tag site.ExampleNews
</rule>
# it is also supported regexp back reference.
<rule>
key domain
pattern /^(mail)\.(example)\.com$/
tag site.$2$1
</rule>
<rule>
key domain
pattern /.+/
tag site.unmatched
</rule>
</match>
<match site.*>
@type mongo
host localhost
database apache_access
remove_tag_prefix site
tag_mapped
capped
capped_size 100m
</match>
<match clear>
@type null
</match>
capitalize_regex_backreference
是否大写正则匹配后项引用项的首字母。
默认false,不大写。
<rule>配置项
设置匹配及重写规则。
key:指定日志记录中的匹配字段
pattern:匹配规则使用的正则表达式
tag:新的tag。
支持正则表达式的后向引用,参加上例中第六个rule。
支持以下占位符:
${tag} 或 __TAG__:原tag
${tag_parts[n]} 或 __TAG_PARTS[n]__:取原tag的第n个字段
${hostname} 或 __HOSTNAME__:主机名
invert:默认为false。
true表示若匹配失败,则重写tag;
false表示匹配成功时才重写tag。
占位符参数:
remove_tag_prefix:移除原tag中的前缀
remove_tag_regexp:移除原tag中的正则匹配部分
hostname_command:设置hostname使用的命令。
默认使用hostname获取完整的主机名。
可使用hostname -s获取较短的主机名。
# built-in TCP input
<source>
@type forward
</source>
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
@type rewrite_tag_filter
<rule>
key status
pattern /^(?!404)$/
tag clear
</rule>
<rule>
key path
pattern /.+/
tag mongo.apache.access.error404
</rule>
</match>
# Store deadlinks log into mongoDB
<match mongo.apache.access.error404>
@type mongo
host 10.100.1.30
database apache
collection deadlinks
capped
capped_size 50m
</match>
# Clear tag
<match clear>
@type null
</match>
<match app.**>
@type rewrite_tag_filter
<rule>
key level
pattern /(.+)/
tag app.$1
</rule>
</match>
<match app.**>
@type forward
# ...
</match>
赞 (0)