GitHub user Susmit07 edited a comment on the discussion: Pekko Connectors for pulling data from HDFS to putting into S3
> If you need dynamic scaling, you could consider creating a Pekko Cluster > using [Cluster > Sharding](https://pekko.apache.org/docs/pekko/current/typed/cluster-sharding.html). The connector should be able to pull data from hdfs and ceph (s3 like object store) the file format will be parquet. For better scalability and parallelism we are thinking to go deploy the application code within 2 K8s pods (to start with). https://nightlies.apache.org/pekko/docs/pekko-management/main-snapshot/docs/kubernetes-deployment/forming-a-cluster.html is it the correct document to point to or the approach for dynamic scaling is wrong Lastly thanks for the replies and the discussion GitHub link: https://github.com/apache/pekko-connectors/discussions/814#discussioncomment-10712032 ---- This is an automatically sent email for notifications@pekko.apache.org. To unsubscribe, please send an email to: notifications-unsubscr...@pekko.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@pekko.apache.org For additional commands, e-mail: notifications-h...@pekko.apache.org
