Spark 1.5.2 + Hive 1.0.0 in Amazon EMR 4.2.0

], HttpClientSendRequestTime=[0.145], 15/11/30 21:40:21 WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.TApplication*Exception: Invalid method name: 'alter_table_with_cascade'* Thanks! -- *Daniel Lopes, B.Eng* Data Scientist - BankFacil CREA/SP

Re: Scala VS Java VS Python

uage for spark examples. > > Thank for the advice > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- *Daniel Lopes, B.Eng* Data Scientist - BankFacil CREA/SP 5069410560 <http://edital.confea.org.br

spark-csv on Amazon EMR

Hi, Some know how to use spark-csv in create-cluster statement of Amazon EMR CLI? Best, -- *Daniel Lopes, B.Eng* Data Scientist - BankFacil CREA/SP 5069410560 <http://edital.confea.org.br/ConsultaProfissional/cartao.aspx?rnp=2613651334> Mob +55 (18) 99764-2733 <callto:+551899764273

Re: UDF with 2 arguments

), what's the version of Spark you have? > > >>> from pyspark.sql.functions import udf > >>> def f(a, b): pass > ... > >>> my_udf = udf(f) > >>> from pyspark.sql.types import * > >>> my_udf = udf(f, IntegerType()) > > &

Re: unsubscribe)

Hi Uzi, To unsubscribe e-mail: user-unsubscr...@spark.apache.org *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?pk_campaign=EmailSignature_kwd=daniel-lopes> On Mon,

Check out Kyper! Trying to be Uber of Data

I just signed up for Kyper and thought you might be interested, too! http://l.aunch.us/L7Ezb

Re: unsubscribe

please send to user-unsubscr...@spark.apache.org *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lopes> On Tue, Aug 2, 2016 at 10

Re: year out of range

| +-++-++--+---+-+-+--+--++-+-+--+++++--+++--+-+--+ - *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com

Re: year out of range

Thanks, I *tested* the function offline and works Tested too with select * from after convert the data and see the new data good *but* if I *register as temp table* to *join other table* stilll shows *the same error*. ValueError: year out of range Best, *Daniel Lopes* Chief Data and Analytics

year out of range

", line 1563, in func = lambda _, it: map(lambda x: returnType.toInternal(f(*x)), it) File "/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/sql/types.py", line 191, in toInternal else time.mktime(dt.timetuple())) *ValueError: year out of range *

Re: year out of range

Thanks Mike, A good way to debug! Was that already! Best, *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lopes> On Thu, Sep 8

Re: year out of range

Thanks Ayan! *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lopes> On Thu, Sep 8, 2016 at 7:54 PM, ayan guha <guha.a...@

Spark + Parquet + IBM Block Storage at Bluemix

: Lost task 60.9 in stage 30.0 (TID 2556, yp-spark-dal09-env5-0039): org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException:* Missing mandatory configuration option: fs.swift.service.keystone.auth.url* at org.apache.hadoop.fs.swift.http.RestClientBindings.copy(RestClientBindings.java:223) at org.apache.hadoop.fs.swift.http.RestClientBindings.bind(RestClientBindings.java:147) *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lopes>

Re: Spark + Parquet + IBM Block Storage at Bluemix

Thanks Steve, But this error occurs only with parquet files, CSVs works. Best, *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lo

Re: unsubscribe

Hi Chang, just send a e-mail to user-unsubscr...@spark.apache.org Best, *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lopes>

Re: Fw: Spark + Parquet + IBM Block Storage at Bluemix

Hi Mario, Thanks for your help, so I will keeping using CSVs Best, *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lopes> On Mo

Re: unsubscribe

To unsubscribe e-mail: user-unsubscr...@spark.apache.org *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | http://www.daniellopes.com.br www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature_term=daniel-lopes> On Mon, Sep 26, 2016 at