Add Kinesis Binder for Spring Cloud Stream #1520
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
artembilan
requested review from
MatejNedic,
maciejwalkowiak and
tomazfernandes
as code owners
github-actions
bot
added
type: documentation
Contributor
Author
|
I've just added DynamoDB Streams support over Kinesis. Another question: what Maven goal should I call to reformat the code properly? Thanks |
Member
|
Hey, we can keep it in a same PR it is fine. Will review it by end of the weekend. you can run -> mvn spotless::apply Tnx on PR |
* Introduce `spring-cloud-aws-kinesis-stream-binder` module (the name could be changed) * Mention it in the root POM * Manage it as a dependency * Also, manage `spring-cloud-stream-dependencies` * Add docs, include an image of Binder architecture
* Add `com.amazonaws:dynamodb-streams-kinesis-adapter` dependency which provides and adapter from DynamoDB Streams Client to Kinesis Client. That also incudes some convenient API for KCL configuration. * Add `SpringDynamoDBStreamsAdapterClient` extension to mimic Kinesis API required for the `KinesisMessageDrivenChannelAdapter` * Support DynamoDB Streams Adapter configuration in the `KclMessageDrivenChannelAdapter` * Add tests to verify DynamoDB Stream consumption via both `KinesisMessageDrivenChannelAdapter` and `KclMessageDrivenChannelAdapter` * Improve other KCL tests for race conditions and timing * Implement DynamoDB Streams support in the Kinesis Binder * Include `dynamodb-streams-kinesis-adapter` dependency into `spring-cloud-aws-starter-integration-kinesis-client` since `KclMessageDrivenChannelAdapter` now depends on that adapter API. * Fix "shard iterator ready" race condition in the `KinesisBinderFunctionalTests` Related to: spring-cloud/spring-cloud-stream-binder-aws-kinesis#205
* Adapt to the latest dependencies, including new Spring Boot deps for tracing * Fix `KinesisBinderObservationTests` according to the latest Spring Boot * Use Spring Cloud Stream `5.0.0` * Disable `KinesisBinderFunctionalTests` since there is something off with Spring Cloud Stream and batch handling
artembilan
force-pushed
the
kinesis-binder
branch
from
dcabde1 to
fe0d577
Compare
for the steam to be enabled on the table before performing data manipulation
Looks like the batch consumer for `Message<List<Message<?>>>` cannot be deduced yet. * Use plain `Consumer<Message<?>>` for the `Message<List<Message<?>>>` from the Kinesis binder * Fix typos in docs * Optimize some code flows
in attempt to mitigate race conditions * Add WARN for received CDC events in the `DynamoDbStreamToKinesisIntegrationTests`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
spring-cloud-aws-kinesis-stream-bindermodule (the name could be changed)spring-cloud-stream-dependencies📢 Type of change
📜 Description
💡 Motivation and Context
💚 How did you test it?
📝 Checklist
🔮 Next steps