[jira] [Created] (SPARK-20622) Parquet partition discovery for non key=value named directories
Noam Asor created SPARK-20622:
-
Summary: Parquet partition discovery for non key=value named
directories
Key: SPARK-20622
URL: https://issues.apache.org/jira/browse/SPARK-20622
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.2.0
Reporter: Noam Asor
h4. Why
There are cases where traditional M/R jobs and RDD based Spark jobs writes out
partitioned parquet in 'value only' named directories i.e.
{{hdfs:///some/base/path/2017/05/06}} and not in 'key=value' named directories
i.e. {{hdfs:///some/base/path/year=2017/month=05/day=06}} which prevents users
from leveraging Spark SQL parquet partition discovery when reading the former
back.
h4. What
This issue is a proposal for a solution which will allow Spark SQL to discover
parquet partitions for 'value only' named directories.
h4. how
By introducing a new Spark SQL read option *partitionTemplate*.
*partitionTemplate* is in a Path form and it should include base path followed
by the missing 'key=' as a template for transforming 'value only' named dirs to
'key=value' named dirs. In the example above this will look like:
{{hdfs:///some/base/path/year=/month=/day=/}}.
To simplify the solution this option should be tied with *basePath* option,
meaning that *partitionTemplate* option is valid only if *basePath* is set also.
In the end for the above scenario, this will look something like:
{code}
spark.read
.option("basePath", "hdfs:///some/base/path")
.option("basePath", "hdfs:///some/base/path/year=/month=/day=/")
.parquet(...)
{code}
which will allow Spark SQL to do parquet partition discovery on the following
directory tree:
{code}
some
|--base
|--path
|--2016
|--...
|--2017
|--01
|--02
|--...
|--15
|--...
|--...
{code}
adding to the schema of the resulted DataFrame the columns year, month, day and
their respective values as expected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20622) Parquet partition discovery for non key=value named directories
[
https://issues.apache.org/jira/browse/SPARK-20622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noam Asor updated SPARK-20622:
--
Description:
h4. Why
There are cases where traditional M/R jobs and RDD based Spark jobs writes out
partitioned parquet in 'value only' named directories i.e.
{{hdfs:///some/base/path/2017/05/06}} and not in 'key=value' named directories
i.e. {{hdfs:///some/base/path/year=2017/month=05/day=06}} which prevents users
from leveraging Spark SQL parquet partition discovery when reading the former
back.
h4. What
This issue is a proposal for a solution which will allow Spark SQL to discover
parquet partitions for 'value only' named directories.
h4. How
By introducing a new Spark SQL read option *partitionTemplate*.
*partitionTemplate* is in a Path form and it should include base path followed
by the missing 'key=' as a template for transforming 'value only' named dirs to
'key=value' named dirs. In the example above this will look like:
{{hdfs:///some/base/path/year=/month=/day=/}}.
To simplify the solution this option should be tied with *basePath* option,
meaning that *partitionTemplate* option is valid only if *basePath* is set also.
In the end for the above scenario, this will look something like:
{code}
spark.read
.option("basePath", "hdfs:///some/base/path")
.option("partitionTemplate", "hdfs:///some/base/path/year=/month=/day=/")
.parquet(...)
{code}
which will allow Spark SQL to do parquet partition discovery on the following
directory tree:
{code}
some
|--base
|--path
|--2016
|--...
|--2017
|--01
|--02
|--...
|--15
|--...
|--...
{code}
adding to the schema of the resulted DataFrame the columns year, month, day and
their respective values as expected.
was:
h4. Why
There are cases where traditional M/R jobs and RDD based Spark jobs writes out
partitioned parquet in 'value only' named directories i.e.
{{hdfs:///some/base/path/2017/05/06}} and not in 'key=value' named directories
i.e. {{hdfs:///some/base/path/year=2017/month=05/day=06}} which prevents users
from leveraging Spark SQL parquet partition discovery when reading the former
back.
h4. What
This issue is a proposal for a solution which will allow Spark SQL to discover
parquet partitions for 'value only' named directories.
h4. how
By introducing a new Spark SQL read option *partitionTemplate*.
*partitionTemplate* is in a Path form and it should include base path followed
by the missing 'key=' as a template for transforming 'value only' named dirs to
'key=value' named dirs. In the example above this will look like:
{{hdfs:///some/base/path/year=/month=/day=/}}.
To simplify the solution this option should be tied with *basePath* option,
meaning that *partitionTemplate* option is valid only if *basePath* is set also.
In the end for the above scenario, this will look something like:
{code}
spark.read
.option("basePath", "hdfs:///some/base/path")
.option("basePath", "hdfs:///some/base/path/year=/month=/day=/")
.parquet(...)
{code}
which will allow Spark SQL to do parquet partition discovery on the following
directory tree:
{code}
some
|--base
|--path
|--2016
|--...
|--2017
|--01
|--02
|--...
|--15
|--...
|--...
{code}
adding to the schema of the resulted DataFrame the columns year, month, day and
their respective values as expected.
> Parquet partition discovery for non key=value named directories
> ---
>
> Key: SPARK-20622
> URL: https://issues.apache.org/jira/browse/SPARK-20622
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Noam Asor
>
> h4. Why
> There are cases where traditional M/R jobs and RDD based Spark jobs writes
> out partitioned parquet in 'value only' named directories i.e.
> {{hdfs:///some/base/path/2017/05/06}} and not in 'key=value' named
> directories i.e. {{hdfs:///some/base/path/year=2017/month=05/day=06}} which
> prevents users from leveraging Spark SQL parquet partition discovery when
> reading the former back.
> h4. What
> This issue is a proposal for a solution which will allow Spark SQL to
> discover parquet partitions for 'value only' named directories.
> h4. How
> By introducing a new Spark SQL read option *partitionTemplate*.
> *partitionTemplate* is in a Path form and it should include base path
> followed by the missing 'key=' as a template for transforming 'value only'
> named dirs to 'key=value' named dirs. In the example above this will look
> lik
[jira] [Updated] (SPARK-20622) Parquet partition discovery for non key=value named directories
[
https://issues.apache.org/jira/browse/SPARK-20622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noam Asor updated SPARK-20622:
--
Priority: Minor (was: Major)
> Parquet partition discovery for non key=value named directories
> ---
>
> Key: SPARK-20622
> URL: https://issues.apache.org/jira/browse/SPARK-20622
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Noam Asor
>Priority: Minor
>
> h4. Why
> There are cases where traditional M/R jobs and RDD based Spark jobs writes
> out partitioned parquet in 'value only' named directories i.e.
> {{hdfs:///some/base/path/2017/05/06}} and not in 'key=value' named
> directories i.e. {{hdfs:///some/base/path/year=2017/month=05/day=06}} which
> prevents users from leveraging Spark SQL parquet partition discovery when
> reading the former back.
> h4. What
> This issue is a proposal for a solution which will allow Spark SQL to
> discover parquet partitions for 'value only' named directories.
> h4. How
> By introducing a new Spark SQL read option *partitionTemplate*.
> *partitionTemplate* is in a Path form and it should include base path
> followed by the missing 'key=' as a template for transforming 'value only'
> named dirs to 'key=value' named dirs. In the example above this will look
> like:
> {{hdfs:///some/base/path/year=/month=/day=/}}.
> To simplify the solution this option should be tied with *basePath* option,
> meaning that *partitionTemplate* option is valid only if *basePath* is set
> also.
> In the end for the above scenario, this will look something like:
> {code}
> spark.read
> .option("basePath", "hdfs:///some/base/path")
> .option("partitionTemplate", "hdfs:///some/base/path/year=/month=/day=/")
> .parquet(...)
> {code}
> which will allow Spark SQL to do parquet partition discovery on the following
> directory tree:
> {code}
> some
> |--base
>|--path
> |--2016
> |--...
> |--2017
>|--01
>|--02
>|--...
>|--15
>|--...
>|--...
> {code}
> adding to the schema of the resulted DataFrame the columns year, month, day
> and their respective values as expected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20622) Parquet partition discovery for non key=value named directories
[
https://issues.apache.org/jira/browse/SPARK-20622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040515#comment-16040515
]
Noam Asor commented on SPARK-20622:
---
The provided pull request is not complete and is rather in a POC state.
If it will be useful enough to be looked at and considered as part of Spark
than it should get polished first.
> Parquet partition discovery for non key=value named directories
> ---
>
> Key: SPARK-20622
> URL: https://issues.apache.org/jira/browse/SPARK-20622
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Noam Asor
>Priority: Minor
>
> h4. Why
> There are cases where traditional M/R jobs and RDD based Spark jobs writes
> out partitioned parquet in 'value only' named directories i.e.
> {{hdfs:///some/base/path/2017/05/06}} and not in 'key=value' named
> directories i.e. {{hdfs:///some/base/path/year=2017/month=05/day=06}} which
> prevents users from leveraging Spark SQL parquet partition discovery when
> reading the former back.
> h4. What
> This issue is a proposal for a solution which will allow Spark SQL to
> discover parquet partitions for 'value only' named directories.
> h4. How
> By introducing a new Spark SQL read option *partitionTemplate*.
> *partitionTemplate* is in a Path form and it should include base path
> followed by the missing 'key=' as a template for transforming 'value only'
> named dirs to 'key=value' named dirs. In the example above this will look
> like:
> {{hdfs:///some/base/path/year=/month=/day=/}}.
> To simplify the solution this option should be tied with *basePath* option,
> meaning that *partitionTemplate* option is valid only if *basePath* is set
> also.
> In the end for the above scenario, this will look something like:
> {code}
> spark.read
> .option("basePath", "hdfs:///some/base/path")
> .option("partitionTemplate", "hdfs:///some/base/path/year=/month=/day=/")
> .parquet(...)
> {code}
> which will allow Spark SQL to do parquet partition discovery on the following
> directory tree:
> {code}
> some
> |--base
>|--path
> |--2016
> |--...
> |--2017
>|--01
>|--02
>|--...
>|--15
>|--...
>|--...
> {code}
> adding to the schema of the resulted DataFrame the columns year, month, day
> and their respective values as expected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-14999) RDDs union checks if the ==
Noam Asor created SPARK-14999: - Summary: RDDs union checks if the == Key: SPARK-14999 URL: https://issues.apache.org/jira/browse/SPARK-14999 Project: Spark Issue Type: Improvement Reporter: Noam Asor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-14999) RDDs union checks if the ==
[ https://issues.apache.org/jira/browse/SPARK-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noam Asor closed SPARK-14999. - Resolution: Invalid > RDDs union checks if the == > --- > > Key: SPARK-14999 > URL: https://issues.apache.org/jira/browse/SPARK-14999 > Project: Spark > Issue Type: Improvement >Reporter: Noam Asor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14999) RDDs union
[ https://issues.apache.org/jira/browse/SPARK-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noam Asor updated SPARK-14999: -- Summary: RDDs union (was: RDDs union checks if the ==) > RDDs union > --- > > Key: SPARK-14999 > URL: https://issues.apache.org/jira/browse/SPARK-14999 > Project: Spark > Issue Type: Improvement >Reporter: Noam Asor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22247) Hive partition filter very slow
[
https://issues.apache.org/jira/browse/SPARK-22247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201971#comment-16201971
]
Noam Asor commented on SPARK-22247:
---
Maybe this issue is related SPARK-17992
> Hive partition filter very slow
> ---
>
> Key: SPARK-22247
> URL: https://issues.apache.org/jira/browse/SPARK-22247
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
>Affects Versions: 2.0.2, 2.1.1
>Reporter: Patrick Duin
>Priority: Minor
>
> I found an issue where filtering partitions using a dataframe results in very
> bad performance.
> To reproduce:
> Create a hive table with a lot of partitions and write a spark query on that
> table that filters based on the partition column.
> In my use case I've got a table with about 30k partitions.
> I filter the partitions using some scala via spark-shell:
> {{table.filter("partition=x or partition=y")}}
> This results in a Hive thrift API call:{{ #get_partitions('db', 'table',
> -1)}} which is very slow (minutes) and loads all metastore partitions in
> memory.
> Doing a more simple filter:
> {{table.filter("partition=x)}}
> Results in a Hive Thrift API call:{{ #get_partitions_by_filter('db', 'table',
> 'partition = "x', -1)}} which is very fast (seconds) and only fetches
> partition X into memory.
> If possible Spark should translate the filter into the more performant Thrift
> call or fallback to a more scalable solution where it filters our partitions
> without having to loading them all into memory first (for instance fetching
> the partitions in batches).
> I've posted my original question on
> [SO|https://stackoverflow.com/questions/46152526/how-should-i-configure-spark-to-correctly-prune-hive-metastore-partitions]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
