Hi Jacek,
The issue might not be very widespread. I couldn’t reproduce it. Can you see if
I am doing anything incorrect in the below queries?
scala> spark.range(10).write.saveAsTable("t1")
scala> spark.sql("describe formatted t1").show(100, false)
++-
Hi Sean,
Thanks again for helping me to remain sane and that the issue is not
imaginary :)
I'd expect to be spark-warehouse in the directory where spark-shell is
executed (which is what has always been used for the metastore).
I'm reviewing all the changes between 2.3.1..2.3.2 to find anything
r
+1 (non-binding)
On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <
stavros.kontopou...@lightbend.com> wrote:
> +1
>
> Stavros
>
> On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen wrote:
>
>> +1, with comments:
>>
>> There are 5 critical issues for 2.4, and no blockers:
>> SPARK-25378 ArrayData.t
Hm, changes in the behavior of the default warehouse dir sound
familiar, but anything I could find was resolved well before 2.3.1
even. I don't know of a change here. What location are you expecting?
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12343289
On Sun,
Hi Sean,
I thought so too, but the path "file:/user/hive/warehouse/" should not have
been used in the first place, should it? I'm running it in spark-shell
2.3.2. Why would there be any changes between 2.3.1 and 2.3.2 that I just
downloaded and one worked fine while the other did not? I had to dow
Removing user@ since cross-posting multiple mailing lists are considered as
not-good practice.
My knowledge is based on the codebase after SPARK-23966, so I'm reading
SPARK-23966 back and try to explain what I can see in the patch. Anyone
please correct me if I'm missing here.
You may want to not
Besides I have tried ANALYZE statement. It has no use cause I need the single
partition but get the total table size by hive parameter 'totalSize' or
'rawSize' and so on
Hi, guys:
I'm using Spark1.6.2.
There are two tables and the small one is a partitioned parquet table;
T
Anyone who can clear doubts on the questions asked here ?
Regards,
Chandan
On Sat, Aug 11, 2018 at 10:03 PM chandan prakash
wrote:
> Hi All,
> I was going through this pull request about new CheckpointFileManager
> abstraction in structured streaming coming in 2.4 :
> https://issues.apache.or
Hi, guys:
I'm using Spark1.6.2.
There are two tables and the small one is a partitioned parquet table;
The total size of the small table is 1000M but each partition only 1M;
When I set spark.sql.autoBroadcastJoinThreshold to 50m and join the two
tables with single partition, I
