SQS or Amazon MQ as previously mentioned are likely to be simpler and more cost-effective than MSK, and will almost certainly be cheaper than Kinesis.The following document details the integration of the Attunity Replicate and Apache kafka tools using Amazon Managed Streaming for Apache Kafka (Amazon MSK), with the aim of real-time data ingestion and consumption from multiple platforms. To be honest, a stand-alone consumer running as a service in your vpc or on-prem is a better idea. You can possibly use Lambda, so long as you cater for failures, timeouts, and other failure modes.
#AWS REDSHIFT EMR MSK SOFTWARE#
You will have to write some sort of bridging software from Kafka -> Other systems It's possible, but it's unlikely there are any turn-key solutions for this. You need an Apache Kafka C++ client, and similar to kinesis above you will need some sort of tunnel or gateway from your on-prem network to the AWS vpc where you have provisioned MSK
You would either need a public Kinesis endpoint, or a private Kinesis endpoint accessible via some sort of tunnel or gateway between your on-prem network and your AWS vpc. If you don't need to worry about ordering, routing, etc it is a persistent and reliable (if clunky) technology that lots of people use to great success.Īmazon publishes a C++ SDK for their services - I would be stunned if there wasn't a Kinesis client as part of this. SQS is a simple api-based queueing system - you publish to an SQS queue, and you consume from the queue. Just from your questions it's clear you have not interacted with Kafka at all, so you're going to have a steep learning curve.
Have you considered rather looking at SQS or Amazon MQ ? Both are considerably simpler to use and manage than Kafka or Kinesis. Kinesis is meant to ingest, transform and process terabytes of moving data. Kinesis is a firehose where you need a straw. To be blunt, your use case sounds simple and 50 messages a second is a very low rate.