What is Amazon Kinesis Firehose?
Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data in data lakes, data stores, and analytics tools. It can capture, convert, and load streaming data on Amazon S3, Amazon Redshift, Amazon Elastic search Service, and Splunk, which enables real-time analytics with business intelligence tools and dashboards that you already use. It is a fully managed service that automatically scales to match the performance of your data and does not require ongoing administration. It can compile, compress, modify, and encrypt data before it is loaded, reducing the amount of storage used on targets and increasing security.
From the AWS Management Console, you can easily create a Firehose delivery stream, configure it with just a few clicks, and start streaming data from hundreds of thousands of data sources that need to be continuously loaded to AWS – all in only in few minutes. Before delivering data to Amazon S3, you can configure your distribution stream to automatically convert incoming data to columns such as Apache Parquet and Apache ORC, for cost-effective storage and analytics.
With Kinesis Data Firehose, you only pay for the amount of data you send through the Service, if applicable, for a data format change. There is no minimum fee or setup cost.
Benefits of Amazon Kinesis Firehose
Easy to use – The Amazon Kinesis Data Firehose provides an easy way to capture, convert, and load streaming data in just a few clicks on the AWS Management Console. You can create a firehose distribution stream, select targets, and start sending real-time data from hundreds of thousands of data sources at once. The Service takes care of stream management, including all the scaling, sharpening, and monitoring of data to be used to load data to your target continuously.
Integrated with data lakes and data stores – Amazon Kinesis Data Firehose can be integrated with Amazon S3, Amazon Redshift, and Amazon Elastic Search Service. From the AWS Management Console, you can point to the Kinesis Data Firehose to the Amazon S3 bucket, Amazon Redshift table, or Amazon elastic search domain. You can use your existing analytics applications and tools to analyze streaming data.
Serverless data conversion – The Amazon Kinesis Data Firehose allows you to prepare your streaming data before loading it into data stores. With the Kinesis Data Firehose, you can easily convert raw streaming data from your data sources into the forms your target data stores need, without creating your own data processing pipelines.
Near real-time – Amazon Kinesis Data Firehose captures and loads data in real-time. Within 60 seconds of sending data to the Service, it loads new data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk. As a result, you can quickly access new data and respond rapidly to business and operational events.
There is no ongoing administration – Amazon Kinesis Data Firehose is a fully managed service that automatically provides, leads, and measures the compute, memory, and network resources needed to load your streaming data. When setting up, Kinesis Data Loader will continue to load data when it arrives.
Pay only for what you use – With Amazon Kinesis Data Firehose, you only pay for the amount of data you send through the service, if applicable, for a data format change. There are no minimum fees or upfront obligations.
How It Works
IoT Analytics – With Amazon’s Kinesis Data Firehose, consumers can continuously capture data from connected devices such as equipment, embedded sensors and TV set-top boxes. Kinesis Data Firehose loads data on Amazon S3 and Amazon Redshift, which enables you to provide your customers with near real-time access to metrics, insights and dashboards.
Click Stream Analytics – The Amazon Kinesis Data Firehose can be used to provide real-time analysis of digital content, enabling authors and marketers to connect with their customers in the most effective way. You can stream billions of short messages delivered to condensed, encrypted and Amazon Reset Search service and Amazon Redshift. From there, you can collect, filter and process data and update content performance dashboards in real-time. For example, Hearst Corporation developed a click-stream analytics platform using the Kinesis Data Firehouse to send and process 30 terabytes of data per day from 300+ websites around the world. With this platform, the entire data stream, from website clicks to integrated measurements, can be made available to the editors within minutes of Headstall.
Log Analytics – Recording data from your applications and servers running in the cloud and on campus will help you monitor your applications and troubleshoot issues faster. For example, identifying the root cause by collecting, monitoring, and analyzing log data can be used to detect when application errors occur. You can easily install and configure the Amazon Kinesis Agent on your servers to automatically view application and server log files and send data to the Kinesis data firehose. Kinesis Data Firehose regularly streams log data to the Amazon Reset Search service, so you can view and analyze data with Kibana.
Security monitoring – Kinesis Data Firehouse supports Splunk as a destination. This means you can capture and transmit network traffic flow records to the Kinesis Data Firehose, which can transfer, enrich, and load data into Splunk. With this solution, you can monitor network security in real-time and alert youself when a potential threat occurs.