Logstash is a Data pipeline it consist of three stages inputs, filters and outputs.
input-> can get data from Kafka,relation database , file or any other input source or can get data from multiple input source also.
filters -> filter what kind of data we need to process.
outlet -> where we need to write data after filter like elastic
Take an example we want to read logs of access.log from file using logstash
like log stash receive one line of log -> then process line using grok pattern then push into elastic .
Having too many concurrent indexing connections may result in a high bulk queue, bad responsiveness and timeouts. And for that reason in most cases, the common setup is to have Logstash placed between Beat instances and Elasticsearch to control the indexing.
And for larger scale system, the common setup is having a buffering message queue (Apache Kafka, Rabbit MQ or Redis) between Beats and Logstash for resilency to avoid congestion on Logstash during event spikes.
No comments:
Post a Comment