把 apache log 同步到 kinesis firehose

使用 Kinesis Agent进行 Amazon Kinesis Firehose 写入操作http://docs.aws.amazon.com/zh_cn/firehose/latest/dev/writing-with-agents.html

步驟
1. 建立 kinesis firehose
2. 利用lambda做資料整理,把 apache log 轉 json,時間格式換成 elasticsearch 能吃的
3.開 aws elasticsearch service
4.安裝Amazon Kinesis Firehose agent自動化，下面是安裝範例

#!/bin/bash

sudo yum install aws-kinesis-agent -y

sudo usermod -a -G root aws-kinesis-agent-user

sudo chmod g+rx /var/log/httpd

setting='{

"cloudwatch.emitMetrics": true,

"firehose.endpoint": "https://firehose.us-east-1.amazonaws.com",

"flows": [

{

"filePattern": "/var/log/httpd/error_log*",

"deliveryStream": "apache_error_log"

}

]

}' ;

echo $setting > /etc/aws-kinesis/agent.json

sudo service aws-kinesis-agent start

sudo chkconfig aws-kinesis-agent on

看 log /var/log/aws-kinesis-agent/aws-kinesis-agent.log

--------------------
#aws #elasticsearch #firehose #kinesis