Re: Loading lots of parquet files into dataframe from s3

You can do something like this: ObjectListing objectListing; do { objectListing = s3Client.listObjects(listObjectsRequest); for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) { if

Re: Submitting Spark Applications using Spark Submit

Hi, To make the jar files as part of the jar which you would like to use, you should create a uber jar. Please refer to the following: https://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html -- View this message in context:

Read multiple files from S3

Hi, I am trying to get a collection of files according to LastModifiedDate from S3 List String FileNames = new ArrayListString(); ListObjectsRequest listObjectsRequest = new ListObjectsRequest() .withBucketName(s3_bucket) .withPrefix(logs_dir);