ログってなんぼ

日々のメモです

td-agentを使ってMongoDBにApacheのアクセスログを保存

td-agentとかruby1.9.2 or heigherとかgemとかインストール済み前提で。

td-agent経由でTDへデータを送っていますが、mongodbプラグインを入れてから設定を少し追加して、localhostのMongoDBにもデータを入れてみた。

fluent Mongoプラグインを入れる

[code]

/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mongo

Successfully installed fluent-plugin-mongo-0.7.1 1 gem installed Installing ri documentation for fluent-plugin-mongo-0.7.1... Installing RDoc documentation for fluent-plugin-mongo-0.7.1... [/code]

td-agent.conf

こちらのエントリで作った設定にMongoDBへの保存設定を追加。

[code]

vi /etc/td-agent/td-agent.conf

type tail path /var/log/httpd/kndl.access_log pos_file /tmp/kndl.access.log.pos format apache tag td.test.kndl type copy type tdlog apikey XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX auto_create_table buffer_type file buffer_path /var/log/td-agent/buffer/td use_ssl true flush_interval 10s type mongo database httpd collection accesslog host localhost port 27017 flush_interval 10s [/code]

td-agent再起動

[code]

/etc/init.d/td-agent restart

[/code]

mongo確認

[code]

mongo

MongoDB shell version: 2.4.5 connecting to: test

show dbs httpd 0.203125GB local 0.078125GB mydb 0.203125GB use httpd switched to db httpd show collections accesslog system.indexes db.accesslog.find() { "id" : ObjectId("52137efa72adc02db9000001"), "host" : "5.10.83.51", "user" : "-", "method" : "GET", "path" : "/items/detail/B00BUL5MMY/", "code" : "200", "size" : "21701", "referer" : "-", "agent" : "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)", "time" : ISODate("2013-08-20T13:23:56Z") } { "id" : ObjectId("52137efa72adc02db9000002"), "host" : "66.249.77.174", "user" : "-", "method" : "GET", "path" : "/items/detail/B007EBJEJG", "code" : "200", "size" : "20907", "referer" : "-", "agent" : "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "time" : ISODate("2013-08-20T13:26:29Z") } { "id" : ObjectId("52137efa72adc02db9000003"), "host" : "66.249.77.174", "user" : "-", "method" : "GET", "path" : "/items/detail/B00C36ZNRY", "code" : "200", "size" : "28875", "referer" : "-", "agent" : "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "time" : ISODate("2013-08-20T13:27:41Z") } { "id" : ObjectId("52137efa72adc02db9000004"), "host" : "119.235.237.18", "user" : "-", "method" : "GET", "path" : "/items/search/d/2275256051/1/Kate%20Pocklington", "code" : "200", "size" : "14232", "referer" : "-", "agent" : "Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)", "time" : ISODate("2013-08-20T13:27:43Z") } { "id" : ObjectId("52137efa72adc02db9000005"), "host" : "119.235.237.18", "user" : "-", "method" : "GET", "path" : "/items/search/s/2275256051/1/D.J.%20Currivean", "code" : "200", "size" : "14224", "referer" : "-", "agent" : "Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)", "time" : ISODate("2013-08-20T13:32:15Z") } { "id" : ObjectId("52137efa72adc02db9000006"), "host" : "66.249.77.174", "user" : "-", "method" : "GET", "path" : "/items/detail/B00BB0MSQ2", "code" : "200", "size" : "19573", "referer" : "-", "agent" : "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "time" : ISODate("2013-08-20T13:32:43Z") } { "_id" : ObjectId("52137efa72adc02db9000007"), "host" : "66.249.75.19", "user" : "-", "method" : "GET", "path" : "/items/detail/B009DEMI9A", "code" : "200", "size" : "22115", "referer" : "-", "agent" : "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "time" : ISODate("2013-08-20T13:36:14Z") } [/code]

入った〜!