RE: Apache Spark

2021-01-26 Thread Синий Андрей
Thank you for the information. If necessary, I will write. С уважением, Андрей Синий Руководитель направления Центр управления программным обеспечением Филиал ПАО «МТС» в Нижегородской области Публичное акционерное общество «Мобильные ТелеСистемы» __ IP: 90096 mob:

Unsubscribe

2021-01-26 Thread Dharmin Siddesh J
Unsubscribe

unsubscribe

2021-01-26 Thread heri wijayanto

Re: Apache Spark

2021-01-26 Thread Maxim Gekk
Hi Андрей, You can write to https://databricks.com/company/contact . Probably, we can offer something to you. For instance, Databricks has OEM program which might be interesting to you: https://partners.databricks.com/prm/English/c/Overview Maxim Gekk Software Engineer Databricks, Inc. On

Re: Apache Spark

2021-01-26 Thread Lalwani, Jayesh
All of the major cloud vendors have some sort of Spark offering. They provide support if you build in their cloud. From: Синий Андрей Date: Tuesday, January 26, 2021 at 7:52 AM To: "user@spark.apache.org" Subject: [EXTERNAL] Apache Spark CAUTION: This email originated from outside of the

Re: Apache Spark

2021-01-26 Thread Ivan Petrov
Hello Andrey, you can try to reach Beeline beeline.ru, they use Databricks as far as I know. вт, 26 янв. 2021 г. в 15:01, Sean Owen : > To clarify: Apache projects and the ASF do not provide paid support. > However there are many vendors who provide distributions of Apache Spark > who will

Odd NoClassDefFoundError exception

2021-01-26 Thread Lavelle, Shawn
Hello Spark Community, I have a Spark-SQL problem where I am receiving a NoClassDefFoundError error for org.apache.spark.sql.catalyst.util.RebaseDateTime$ . This happens for any query with a filter on a Timestamp column when the query is first run programmatically but not when the query is

Re: Apache Spark

2021-01-26 Thread Sean Owen
To clarify: Apache projects and the ASF do not provide paid support. However there are many vendors who provide distributions of Apache Spark who will provide technical support - not nearly just Databricks but Cloudera, etc. There are also plenty of consultancies and individuals who can provide

Re: Apache Spark

2021-01-26 Thread Gourav Sengupta
Hi, why do you want to buy paid SPARK? Regards, Gourav On Tue, Jan 26, 2021 at 1:22 PM Pasha Finkelshteyn < pavel.finkelsht...@gmail.com> wrote: > Hi Andrey, > > It looks like you may contact Databricks for that. > Also it would be easier for non-russian spaekers to respond you if your > name

Re: Apache Spark

2021-01-26 Thread Pasha Finkelshteyn
Hi Andrey, It looks like you may contact Databricks for that. Also it would be easier for non-russian spaekers to respond you if your name would be written in English. On 21/01/26 12:41PM, Синий Андрей wrote: > Hello! > > We plan to use Apache Spark software in our organization, can I purchase

Apache Spark

2021-01-26 Thread Синий Андрей
Hello! We plan to use Apache Spark software in our organization, can I purchase paid technical support for this software? С уважением, Андрей Синий Руководитель направления Центр управления программным обеспечением Филиал ПАО «МТС» в Нижегородской области Публичное акционерное общество

Re: Thread spilling sort issue with single task

2021-01-26 Thread German Schiavon
Well if your data is skewed I don't think it can be avoided but mitigated using skew techniques. I'd recommend you to take a look at "salted join" maybe. On Tue, 26 Jan 2021 at 11:29, rajat kumar wrote: > Hi , > > Yes I understand its skew based problem but how can it be avoided . Could >

Re: Thread spilling sort issue with single task

2021-01-26 Thread rajat kumar
Hi , Yes I understand its skew based problem but how can it be avoided . Could you please suggest? I am in Spark2.4 Thanks Rajat On Tue, Jan 26, 2021 at 3:58 PM German Schiavon wrote: > Hi, > > One word : SKEW > > It seems the classic skew problem, you would have to apply skew techniques >

Re: Thread spilling sort issue with single task

2021-01-26 Thread German Schiavon
Hi, One word : SKEW It seems the classic skew problem, you would have to apply skew techniques to repartition your data properly or if you are in spark 3.0+ try the skewJoin optimization. On Tue, 26 Jan 2021 at 11:20, rajat kumar wrote: > Hi Everyone, > > I am running a spark application

Thread spilling sort issue with single task

2021-01-26 Thread rajat kumar
Hi Everyone, I am running a spark application where I have applied 2 left joins. 1st join in Broadcast and another one is normal. Out of 200 tasks , last 1 task is stuck . It is running at "ANY" Locality level. It seems data skewness issue. It is doing too much spill and shuffle write is too