Re: Exposing JIRA issue types at GitHub PRs

2019-06-14 Thread Dongjoon Hyun
Now, you can see the exposed component labels (ordered by the number of PRs) here and click the component to search. https://github.com/apache/spark/labels?sort=count-desc Dongjoon. On Fri, Jun 14, 2019 at 1:15 AM Dongjoon Hyun wrote: > Hi, All. > > JIRA and PR is ready for reviews. > >

Re: Filter cannot be pushed via a Join

2019-06-14 Thread William Wong
Hi all, Appreciate any expert may help on this strange behavior.. It is interesting that... I implemented a custom rule to remove empty LocalRelation children under Union and run the same query. The filter 'id = 'a' is inferred to the table2 and pushed via the Join. scala> spark2.sql("SELECT *

Filter cannot be pushed via a Join

2019-06-14 Thread William Wong
Dear all, I created two tables. scala> spark.sql("CREATE TABLE IF NOT EXISTS table1(id string, val string) USING PARQUET"); 19/06/14 23:49:10 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0

Spark Kafka Streaming stopped

2019-06-14 Thread Amit Sharma
we are using spark kafka streaming. We have 6 nodes in kafka cluster if any of the node is getting down we are getting below exception and streaming stopped. ERROR DirectKafkaInputDStream:70 - ArrayBuffer(kafka.common.NotLeaderForPartitionException, kafka.common.NotLeaderForPartitionException,

[pyspark 2.3+] CountDistinct

2019-06-14 Thread Rishi Shah
Hi All, Recently we noticed that countDistinct on a larger dataframe doesn't always return the same value. Any idea? If this is the case then what is the difference between countDistinct & approx_count_distinct? -- Regards, Rishi Shah

Re: Exposing JIRA issue types at GitHub PRs

2019-06-14 Thread Dongjoon Hyun
Hi, All. JIRA and PR is ready for reviews. https://issues.apache.org/jira/browse/SPARK-28051 (Exposing JIRA issue component types at GitHub PRs) https://github.com/apache/spark/pull/24871 Bests, Dongjoon. On Thu, Jun 13, 2019 at 10:48 AM Dongjoon Hyun wrote: > Thank you for the feedbacks