help on buiding Kudu Java Client 0.7.0

2016-03-01 Thread Darren Hoo 
I have installed the exact version 2.6.1 of protoc

$ which protoc
/usr/local/bin/protoc

$ protoc --version
libprotoc 2.6.1

but when I try to build the Java Client, I got this error:


[INFO] Kudu ... SUCCESS [  1.375 s]
[INFO] Kudu Annotations ... SUCCESS [  0.511 s]
[INFO] Kudu Java Client ... FAILURE [  0.407 s]
[INFO] Kudu's MapReduce bindings .. SKIPPED
[INFO] Collection of tools that interact directly with Kudu SKIPPED
[INFO] Kudu Spark Bindings  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 2.520 s
[INFO] Finished at: 2016-03-01T17:24:18+08:00
[INFO] Final Memory: 35M/1932M
[INFO] 
[ERROR] Failed to execute goal
com.google.protobuf.tools:maven-protoc-plugin:0.1.10:compile (default)
on project kudu-client: protoc failed to execute because: null:
IllegalArgumentException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
execute goal com.google.protobuf.tools:maven-protoc-plugin:0.1.10:compile
(default) on project kudu-client: protoc failed to execute because:
null
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoFailureException: protoc failed
to execute because: null
at 
com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:175)
at 
com.google.protobuf.maven.ProtocCompileMojo.execute(ProtocCompileMojo.java:21)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:70)
at com.google.protobuf.maven.Protoc$Builder.addProtoPathElement(Protoc.java:191)
at 
com.google.protobuf.maven.Protoc$Builder.addProtoPathElements(Protoc.java:201)
at 
com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:157)
... 23 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]


what's the problem?

Re: help on buiding Kudu Java Client 0.7.0

2016-03-01 Thread Mike Percy 
What platform are you on? Are you building from the Git tag or from the
source tarball? Have you built thirdparty already?

Thanks,
Mike

On Tue, Mar 1, 2016 at 11:30 AM, Darren Hoo  wrote:

> I have installed the exact version 2.6.1 of protoc
>
> $ which protoc
> /usr/local/bin/protoc
>
> $ protoc --version
> libprotoc 2.6.1
>
> but when I try to build the Java Client, I got this error:
>
>
> [INFO] Kudu ... SUCCESS [
> 1.375 s]
> [INFO] Kudu Annotations ... SUCCESS [
> 0.511 s]
> [INFO] Kudu Java Client ... FAILURE [
> 0.407 s]
> [INFO] Kudu's MapReduce bindings .. SKIPPED
> [INFO] Collection of tools that interact directly with Kudu SKIPPED
> [INFO] Kudu Spark Bindings  SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 2.520 s
> [INFO] Finished at: 2016-03-01T17:24:18+08:00
> [INFO] Final Memory: 35M/1932M
> [INFO]
> 
> [ERROR] Failed to execute goal
> com.google.protobuf.tools:maven-protoc-plugin:0.1.10:compile (default)
> on project kudu-client: protoc failed to execute because: null:
> IllegalArgumentException -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> execute goal com.google.protobuf.tools:maven-protoc-plugin:0.1.10:compile
> (default) on project kudu-client: protoc failed to execute because:
> null
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoFailureException: protoc failed
> to execute because: null
> at
> com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:175)
> at
> com.google.protobuf.maven.ProtocCompileMojo.execute(ProtocCompileMojo.java:21)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:70)
> at
> com.google.protobuf.maven.Protoc$Builder.addProtoPathElement(Protoc.java:191)
> at
> com.google.protobuf.maven.Protoc$Builder.addProtoPathElements(Protoc.java:201)
> at
> com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:157)
> ... 23 more
> [ERROR]
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
>
>
> what's the problem?
>

Re: help on buiding Kudu Java Client 0.7.0

2016-03-01 Thread Jean-Daniel Cryans 
Hi Darren,

This was fixed in
https://github.com/cloudera/kudu/commit/7a0244c8c539dd800b7269c32a6826d2fdad43d9

If you can't apply the patch, the workaround is to build the third parties.

J-D

On Tue, Mar 1, 2016 at 1:30 AM, Darren Hoo  wrote:

> I have installed the exact version 2.6.1 of protoc
>
> $ which protoc
> /usr/local/bin/protoc
>
> $ protoc --version
> libprotoc 2.6.1
>
> but when I try to build the Java Client, I got this error:
>
>
> [INFO] Kudu ... SUCCESS [
> 1.375 s]
> [INFO] Kudu Annotations ... SUCCESS [
> 0.511 s]
> [INFO] Kudu Java Client ... FAILURE [
> 0.407 s]
> [INFO] Kudu's MapReduce bindings .. SKIPPED
> [INFO] Collection of tools that interact directly with Kudu SKIPPED
> [INFO] Kudu Spark Bindings  SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 2.520 s
> [INFO] Finished at: 2016-03-01T17:24:18+08:00
> [INFO] Final Memory: 35M/1932M
> [INFO]
> 
> [ERROR] Failed to execute goal
> com.google.protobuf.tools:maven-protoc-plugin:0.1.10:compile (default)
> on project kudu-client: protoc failed to execute because: null:
> IllegalArgumentException -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> execute goal com.google.protobuf.tools:maven-protoc-plugin:0.1.10:compile
> (default) on project kudu-client: protoc failed to execute because:
> null
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoFailureException: protoc failed
> to execute because: null
> at
> com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:175)
> at
> com.google.protobuf.maven.ProtocCompileMojo.execute(ProtocCompileMojo.java:21)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:70)
> at
> com.google.protobuf.maven.Protoc$Builder.addProtoPathElement(Protoc.java:191)
> at
> com.google.protobuf.maven.Protoc$Builder.addProtoPathElements(Protoc.java:201)
> at
> com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:157)
> ... 23 more
> [ERROR]
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
>
>
> what's the problem?
>

Re: help on buiding Kudu Java Client 0.7.0

2016-03-01 Thread Darren Hoo 
Thanks to Mike and Jean for your tip, I have built the java client now.

But I still have some difficulties building third parties which I bypassed
before building java client, especially when build_llvm:

-- Looking for __atomic_fetch_add_4 in atomic
-- Looking for __atomic_fetch_add_4 in atomic - not found
CMake Error at cmake/modules/CheckAtomic.cmake:36 (message):
  Host compiler appears to require libatomic, but cannot find it.
Call Stack (most recent call first):
  cmake/config-ix.cmake:291 (include)
  CMakeLists.txt:360 (include)

-- Configuring incomplete, errors occurred!

I am using the 0.7.0 kudu release source tarball, and my platform is CentOS
6.6

$  gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla
--enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada
--enable-java-awt=gtk --disable-dssi
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
--enable-libgcj-multifile --enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
--build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)

$ ls /usr/lib64/libatomic* -hla

lrwxrwxrwx 1 root root  18 Mar  1 21:21 */usr/lib64/libatomic.so.1* ->
*libatomic.so.1.1.0*

-rwxr-xr-x 1 root root 24K Jul 24  2015 */usr/lib64/libatomic.so.1.1.0*



I googled around for a while but did not find any solutions. Any ideas?

Spark SQL on kudu can not contains nullable columns?

2016-03-01 Thread Darren Hoo 
Spark SQL on kudu can not contains nullable columns?

I've create one table in kudu(0.6.0) which has nullable columns,
when I try to use spark sql (using kudu java client 0.7.0) like this:

sqlContext.load("org.kududb.spark",Map("kudu.table" -> "contents",
"kudu.master" -> "master1:7051")).registerTempTable("contents")
sqlContext.sql("SELECT * FROM * FROM contents limit 10").collectAsList()

I got this error:

16/03/02 00:45:42 INFO DAGScheduler: Job 4 failed: collect at :20,
took 11.813423 s
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage
7.0 (TID 62, slave29): java.lang.IllegalArgumentException: The requested
column (4)  is null
at org.kududb.client.RowResult.checkNull(RowResult.java:475)
at org.kududb.client.RowResult.getString(RowResult.java:321)
at org.kududb.client.RowResult.getString(RowResult.java:308)
at org.kududb.spark.KuduRelation.org
$kududb$spark$KuduRelation$$getKuduValue(DefaultSource.scala:144)
at
org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
at
org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at
org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:126)
at
org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:124)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to
(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Is this due to the version incompatibily between my kudu server(0.6.0) and
java client (0.7.0)?

Re: help on buiding Kudu Java Client 0.7.0

2016-03-01 Thread Jean-Daniel Cryans 
See step 2: http://getkudu.io/docs/installation.html#rhel_from_source

Patching the client might just be easier than building all of thirdparties.

J-D

On Tue, Mar 1, 2016 at 9:10 AM, Darren Hoo  wrote:

> Thanks to Mike and Jean for your tip, I have built the java client now.
>
> But I still have some difficulties building third parties which I bypassed
> before building java client, especially when build_llvm:
>
> -- Looking for __atomic_fetch_add_4 in atomic
> -- Looking for __atomic_fetch_add_4 in atomic - not found
> CMake Error at cmake/modules/CheckAtomic.cmake:36 (message):
>   Host compiler appears to require libatomic, but cannot find it.
> Call Stack (most recent call first):
>   cmake/config-ix.cmake:291 (include)
>   CMakeLists.txt:360 (include)
>
> -- Configuring incomplete, errors occurred!
>
> I am using the 0.7.0 kudu release source tarball, and my platform is
> CentOS 6.6
>
> $  gcc -v
> Using built-in specs.
> Target: x86_64-redhat-linux
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/usr/share/info --with-bugurl=
> http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared
> --enable-threads=posix --enable-checking=release --with-system-zlib
> --enable-__cxa_atexit --disable-libunwind-exceptions
> --enable-gnu-unique-object
> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> --enable-java-awt=gtk --disable-dssi
> --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
> --enable-libgcj-multifile --enable-java-maintainer-mode
> --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
> --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
> --build=x86_64-redhat-linux
> Thread model: posix
> gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
>
> $ ls /usr/lib64/libatomic* -hla
>
> lrwxrwxrwx 1 root root  18 Mar  1 21:21 */usr/lib64/libatomic.so.1* ->
> *libatomic.so.1.1.0*
>
> -rwxr-xr-x 1 root root 24K Jul 24  2015 */usr/lib64/libatomic.so.1.1.0*
>
>
>
> I googled around for a while but did not find any solutions. Any ideas?
>
>

Re: Spark SQL on kudu can not contains nullable columns?

2016-03-01 Thread Jean-Daniel Cryans 
Ha yeah that's a good one. I opened this jira:
https://issues.apache.org/jira/browse/KUDU-1360

Basically we forgot to check for nulls :)

J-D

On Tue, Mar 1, 2016 at 9:18 AM, Darren Hoo  wrote:

> Spark SQL on kudu can not contains nullable columns?
>
> I've create one table in kudu(0.6.0) which has nullable columns,
> when I try to use spark sql (using kudu java client 0.7.0) like this:
>
> sqlContext.load("org.kududb.spark",Map("kudu.table" -> "contents",
> "kudu.master" -> "master1:7051")).registerTempTable("contents")
> sqlContext.sql("SELECT * FROM * FROM contents limit 10").collectAsList()
>
> I got this error:
>
> 16/03/02 00:45:42 INFO DAGScheduler: Job 4 failed: collect at
> :20, took 11.813423 s
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage
> 7.0 (TID 62, slave29): java.lang.IllegalArgumentException: The requested
> column (4)  is null
> at org.kududb.client.RowResult.checkNull(RowResult.java:475)
> at org.kududb.client.RowResult.getString(RowResult.java:321)
> at org.kududb.client.RowResult.getString(RowResult.java:308)
> at org.kududb.spark.KuduRelation.org
> $kududb$spark$KuduRelation$$getKuduValue(DefaultSource.scala:144)
> at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
> at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
> at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:126)
> at
> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:124)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at scala.collection.TraversableOnce$class.to
> (TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> Is this due to the version incompatibily between my kudu server(0.6.0) and
> java client (0.7.0)?
>
>

Re: help on buiding Kudu Java Client 0.7.0

2016-03-01 Thread Darren Hoo 
That's great!

Thanks!

On Wed, Mar 2, 2016 at 1:19 AM, Jean-Daniel Cryans  wrote:
> See step 2: http://getkudu.io/docs/installation.html#rhel_from_source
>
> Patching the client might just be easier than building all of thirdparties.
>
> J-D
>
> On Tue, Mar 1, 2016 at 9:10 AM, Darren Hoo  wrote:
>>
>> Thanks to Mike and Jean for your tip, I have built the java client now.
>>
>> But I still have some difficulties building third parties which I bypassed
>> before building java client, especially when build_llvm:
>>
>> -- Looking for __atomic_fetch_add_4 in atomic
>> -- Looking for __atomic_fetch_add_4 in atomic - not found
>> CMake Error at cmake/modules/CheckAtomic.cmake:36 (message):
>>   Host compiler appears to require libatomic, but cannot find it.
>> Call Stack (most recent call first):
>>   cmake/config-ix.cmake:291 (include)
>>   CMakeLists.txt:360 (include)
>>
>> -- Configuring incomplete, errors occurred!
>>
>> I am using the 0.7.0 kudu release source tarball, and my platform is
>> CentOS 6.6
>>
>> $  gcc -v
>> Using built-in specs.
>> Target: x86_64-redhat-linux
>> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
>> --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla
>> --enable-bootstrap --enable-shared --enable-threads=posix
>> --enable-checking=release --with-system-zlib --enable-__cxa_atexit
>> --disable-libunwind-exceptions --enable-gnu-unique-object
>> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
>> --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
>> --enable-libgcj-multifile --enable-java-maintainer-mode
>> --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
>> --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
>> --build=x86_64-redhat-linux
>> Thread model: posix
>> gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
>>
>> $ ls /usr/lib64/libatomic* -hla
>>
>> lrwxrwxrwx 1 root root  18 Mar  1 21:21 /usr/lib64/libatomic.so.1 ->
>> libatomic.so.1.1.0
>>
>> -rwxr-xr-x 1 root root 24K Jul 24  2015 /usr/lib64/libatomic.so.1.1.0
>>
>>
>>
>> I googled around for a while but did not find any solutions. Any ideas?
>>
>>
>
>

Kudu 0.7.0

2016-03-01 Thread Benjamin Kim 
Is the CSD for Cloudera Manager available for Kudu 0.7.0 or can we just add the 
URL to the parcel list?

Thanks,
Ben

Re: Kudu 0.7.0

2016-03-01 Thread Jean-Daniel Cryans 
It will be available very soon. The thing is we (Cloudera) can't start the
binaries release process until the source release has been voted on.

J-D

On Tue, Mar 1, 2016 at 9:42 AM, Benjamin Kim  wrote:

> Is the CSD for Cloudera Manager available for Kudu 0.7.0 or can we just
> add the URL to the parcel list?
>
> Thanks,
> Ben
>
>

Re: Spark SQL on kudu can not contains nullable columns?

2016-03-01 Thread Todd Lipcon 
Perhaps we should target this for 0.7.1 as well, if we're going to do that
follow-up release? Seems like it should be an easy fix (and client-side
only)

-Todd

On Tue, Mar 1, 2016 at 9:29 AM, Jean-Daniel Cryans 
wrote:

> Ha yeah that's a good one. I opened this jira:
> https://issues.apache.org/jira/browse/KUDU-1360
>
> Basically we forgot to check for nulls :)
>
> J-D
>
> On Tue, Mar 1, 2016 at 9:18 AM, Darren Hoo  wrote:
>
>> Spark SQL on kudu can not contains nullable columns?
>>
>> I've create one table in kudu(0.6.0) which has nullable columns,
>> when I try to use spark sql (using kudu java client 0.7.0) like this:
>>
>> sqlContext.load("org.kududb.spark",Map("kudu.table" -> "contents",
>> "kudu.master" -> "master1:7051")).registerTempTable("contents")
>> sqlContext.sql("SELECT * FROM * FROM contents limit 10").collectAsList()
>>
>> I got this error:
>>
>> 16/03/02 00:45:42 INFO DAGScheduler: Job 4 failed: collect at
>> :20, took 11.813423 s
>> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
>> in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage
>> 7.0 (TID 62, slave29): java.lang.IllegalArgumentException: The requested
>> column (4)  is null
>> at org.kududb.client.RowResult.checkNull(RowResult.java:475)
>> at org.kududb.client.RowResult.getString(RowResult.java:321)
>> at org.kududb.client.RowResult.getString(RowResult.java:308)
>> at org.kududb.spark.KuduRelation.org
>> $kududb$spark$KuduRelation$$getKuduValue(DefaultSource.scala:144)
>> at
>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
>> at
>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
>> at
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>> at
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>> at
>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>> at
>> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>> at
>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>> at
>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:126)
>> at
>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:124)
>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>> at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>> at
>> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>> at
>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
>> at
>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
>> at scala.collection.TraversableOnce$class.to
>> (TraversableOnce.scala:273)
>> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
>> at
>> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
>> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
>> at
>> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
>> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
>> at
>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
>> at
>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
>> at
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>> at org.apache.spark.scheduler.Task.run(Task.scala:88)
>> at
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Is this due to the version incompatibily between my kudu server(0.6.0)
>> and java client (0.7.0)?
>>
>>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Spark SQL on kudu can not contains nullable columns?

2016-03-01 Thread Jean-Daniel Cryans 
Yeah didn't think about, are you volunteering Todd? :P I can do it today.

J-D

On Tue, Mar 1, 2016 at 9:57 AM, Todd Lipcon  wrote:

> Perhaps we should target this for 0.7.1 as well, if we're going to do that
> follow-up release? Seems like it should be an easy fix (and client-side
> only)
>
> -Todd
>
> On Tue, Mar 1, 2016 at 9:29 AM, Jean-Daniel Cryans 
> wrote:
>
>> Ha yeah that's a good one. I opened this jira:
>> https://issues.apache.org/jira/browse/KUDU-1360
>>
>> Basically we forgot to check for nulls :)
>>
>> J-D
>>
>> On Tue, Mar 1, 2016 at 9:18 AM, Darren Hoo  wrote:
>>
>>> Spark SQL on kudu can not contains nullable columns?
>>>
>>> I've create one table in kudu(0.6.0) which has nullable columns,
>>> when I try to use spark sql (using kudu java client 0.7.0) like this:
>>>
>>> sqlContext.load("org.kududb.spark",Map("kudu.table" -> "contents",
>>> "kudu.master" -> "master1:7051")).registerTempTable("contents")
>>> sqlContext.sql("SELECT * FROM * FROM contents limit 10").collectAsList()
>>>
>>> I got this error:
>>>
>>> 16/03/02 00:45:42 INFO DAGScheduler: Job 4 failed: collect at
>>> :20, took 11.813423 s
>>> org.apache.spark.SparkException: Job aborted due to stage failure: Task
>>> 0 in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage
>>> 7.0 (TID 62, slave29): java.lang.IllegalArgumentException: The requested
>>> column (4)  is null
>>> at org.kududb.client.RowResult.checkNull(RowResult.java:475)
>>> at org.kududb.client.RowResult.getString(RowResult.java:321)
>>> at org.kududb.client.RowResult.getString(RowResult.java:308)
>>> at org.kududb.spark.KuduRelation.org
>>> $kududb$spark$KuduRelation$$getKuduValue(DefaultSource.scala:144)
>>> at
>>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
>>> at
>>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
>>> at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>>> at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>>> at
>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>>> at
>>> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>>> at
>>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>>> at
>>> scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>>> at
>>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:126)
>>> at
>>> org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:124)
>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>> at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>> at
>>> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>>> at
>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
>>> at
>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
>>> at scala.collection.TraversableOnce$class.to
>>> (TraversableOnce.scala:273)
>>> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
>>> at
>>> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
>>> at
>>> scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
>>> at
>>> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
>>> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
>>> at
>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
>>> at
>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
>>> at
>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>>> at org.apache.spark.scheduler.Task.run(Task.scala:88)
>>> at
>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Is this due to the version incompatibily between my kudu server(0.6.0)
>>> and java client (0.7.0)?
>>>
>>>
>>
>
>
> --
> Todd Lipcon
> Softwa

Re: Spark SQL on kudu can not contains nullable columns?

2016-03-01 Thread Todd Lipcon 
On Tue, Mar 1, 2016 at 10:04 AM, Jean-Daniel Cryans 
wrote:

> Yeah didn't think about, are you volunteering Todd? :P I can do it today.
>

:) I'll volunteer to review!

-Todd


> J-D
>
> On Tue, Mar 1, 2016 at 9:57 AM, Todd Lipcon  wrote:
>
>> Perhaps we should target this for 0.7.1 as well, if we're going to do
>> that follow-up release? Seems like it should be an easy fix (and
>> client-side only)
>>
>> -Todd
>>
>> On Tue, Mar 1, 2016 at 9:29 AM, Jean-Daniel Cryans 
>> wrote:
>>
>>> Ha yeah that's a good one. I opened this jira:
>>> https://issues.apache.org/jira/browse/KUDU-1360
>>>
>>> Basically we forgot to check for nulls :)
>>>
>>> J-D
>>>
>>> On Tue, Mar 1, 2016 at 9:18 AM, Darren Hoo  wrote:
>>>
 Spark SQL on kudu can not contains nullable columns?

 I've create one table in kudu(0.6.0) which has nullable columns,
 when I try to use spark sql (using kudu java client 0.7.0) like this:

 sqlContext.load("org.kududb.spark",Map("kudu.table" -> "contents",
 "kudu.master" -> "master1:7051")).registerTempTable("contents")
 sqlContext.sql("SELECT * FROM * FROM contents limit 10").collectAsList()

 I got this error:

 16/03/02 00:45:42 INFO DAGScheduler: Job 4 failed: collect at
 :20, took 11.813423 s
 org.apache.spark.SparkException: Job aborted due to stage failure: Task
 0 in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage
 7.0 (TID 62, slave29): java.lang.IllegalArgumentException: The requested
 column (4)  is null
 at org.kududb.client.RowResult.checkNull(RowResult.java:475)
 at org.kududb.client.RowResult.getString(RowResult.java:321)
 at org.kududb.client.RowResult.getString(RowResult.java:308)
 at org.kududb.spark.KuduRelation.org
 $kududb$spark$KuduRelation$$getKuduValue(DefaultSource.scala:144)
 at
 org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
 at
 org.kududb.spark.KuduRelation$$anonfun$buildScan$1$$anonfun$apply$1.apply(DefaultSource.scala:126)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at
 scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
 at
 scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
 at
 scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
 at
 scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
 at
 org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:126)
 at
 org.kududb.spark.KuduRelation$$anonfun$buildScan$1.apply(DefaultSource.scala:124)
 at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
 at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
 at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
 at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
 at scala.collection.Iterator$class.foreach(Iterator.scala:727)
 at
 scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
 at
 scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
 at
 scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
 at
 scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
 at scala.collection.TraversableOnce$class.to
 (TraversableOnce.scala:273)
 at scala.collection.AbstractIterator.to(Iterator.scala:1157)
 at
 scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
 at
 scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
 at
 scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
 at
 scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
 at
 org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
 at
 org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
 at
 org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
 at
 org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
 at
 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
 at org.apache.spark.scheduler.Task.run(Task.scala:88)
 at
 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec

Re: Spark on Kudu

2016-03-01 Thread Benjamin Kim 
Hi J-D,

Quick question… Is there an ETA for KUDU-1214? I want to target a version of 
Kudu to begin real testing of Spark against it for our devs. At least, I can 
tell them what timeframe to anticipate.

Just curious,
Benjamin Kim
Data Solutions Architect

[a•mo•bee] (n.) the company defining digital marketing.

Mobile: +1 818 635 2900
3250 Ocean Park Blvd, Suite 200  |  Santa Monica, CA 90405  |  
www.amobee.com

On Feb 24, 2016, at 3:51 PM, Jean-Daniel Cryans 
mailto:jdcry...@apache.org>> wrote:

The DStream stuff isn't there at all. I'm not sure if it's needed either.

The kuduRDD is just leveraging the MR input format, ideally we'd use scans 
directly.

The SparkSQL stuff is there but it doesn't do any sort of pushdown. It's really 
basic.

The goal was to provide something for others to contribute to. We have some 
basic unit tests that others can easily extend. None of us on the team are 
Spark experts, but we'd be really happy to assist one improve the kudu-spark 
code.

J-D

On Wed, Feb 24, 2016 at 3:41 PM, Benjamin Kim 
mailto:bbuil...@gmail.com>> wrote:
J-D,

It looks like it fulfills most of the basic requirements (kudu RDD, kudu 
DStream) in KUDU-1214. Am I right? Besides shoring up more Spark SQL 
functionality (Dataframes) and doing the documentation, what more needs to be 
done? Optimizations?

I believe that it’s a good place to start using Spark with Kudu and compare it 
to HBase with Spark (not clean).

Thanks,
Ben


On Feb 24, 2016, at 3:10 PM, Jean-Daniel Cryans 
mailto:jdcry...@apache.org>> wrote:

AFAIK no one is working on it, but we did manage to get this in for 0.7.0: 
https://issues.cloudera.org/browse/KUDU-1321

It's a really simple wrapper, and yes you can use SparkSQL on Kudu, but it will 
require a lot more work to make it fast/useful.

Hope this helps,

J-D

On Wed, Feb 24, 2016 at 3:08 PM, Benjamin Kim 
mailto:bbuil...@gmail.com>> wrote:
I see this KUDU-1214 targeted for 
0.8.0, but I see no progress on it. When this is complete, will this mean that 
Spark will be able to work with Kudu both programmatically and as a client via 
Spark SQL? Or is there more work that needs to be done on the Spark side for it 
to work?

Just curious.

Cheers,
Ben

Re: Impalad crash

2016-03-01 Thread Casey Ching 
Hi Nick,

We seem to have found the problem. I filed 
https://issues.cloudera.org/browse/IMPALA-3105 about it. The query leads to 
memory corruption so it’s best to avoid it. I couldn’t reproduce the problem on 
Impala’s trunk. I expect this will be fixed in the 0.8 Kudu/Impala release. Let 
me know if you have questions and thanks again for helping with this.

Casey

On February 26, 2016 at 3:36:18 PM, Casey Ching (ca...@cloudera.com) wrote:

Thanks for sending this Nick. Unfortunately it looks like a different issue 
than the previous one — impala::RawValue::Write(void const*, void*, 
impala::ColumnType const&, impala::MemPool*)

I looked at ExprContext::GetValue and didn’t see anything obviously wrong. 
Would you be wiling to upload the core dump? They typically compress well but 
would likely still be a big upload. Alternatively do you have a way to 
reproduce this?

Casey
On February 26, 2016 at 10:47:25 AM, Nick Wolf (nickwo...@gmail.com) wrote:

Enabled core dumps and following is the stacktrace.

#0  0x7f205622ccc9 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56

#1  0x7f20562300d8 in __GI_abort () at abort.c:89

#2  0x7f2055b9e9c5 in os::abort(bool) () from 
/usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#3  0x7f2055d1f607 in VMError::report_and_die() () from 
/usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#4  0x7f2055d1fb8e in crash_handler(int, siginfo*, void*) ()

   from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#5  

#6  0x7f2055b953d1 in os::is_first_C_frame(frame*) () from 
/usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#7  0x7f2055d1dcfd in VMError::report(outputStream*) ()

   from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#8  0x7f2055d1f20a in VMError::report_and_die() () from 
/usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#9  0x7f2055ba38af in JVM_handle_linux_signal () from 
/usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#10 

#11 0x007a453b in impala::ExprContext::GetValue(impala::Expr*, 
impala::TupleRow*) ()

#12 0x00bcf45b in 
impala::UnionNode::EvalAndMaterializeExprs(std::vector > const&, bool, impala::Tuple**, 
impala::RowBatch*) ()

#13 0x00bcf94c in impala::UnionNode::GetNext(impala::RuntimeState*, 
impala::RowBatch*, bool*) ()

#14 0x00bb1ed7 in 
impala::PartitionedAggregationNode::Open(impala::RuntimeState*) ()

#15 0x00b3f7ba in impala::PlanFragmentExecutor::OpenInternal() ()

#16 0x00b3fecd in impala::PlanFragmentExecutor::Open() ()

#17 0x009c7d3a in impala::FragmentMgr::FragmentExecState::Exec() ()

#18 0x009c32dd in 
impala::FragmentMgr::FragmentExecThread(impala::FragmentMgr::FragmentExecState*)
 ()

#19 0x00a482bd in impala::Thread::SuperviseThread(std::string const&, 
std::string const&, boost::function, impala::Promise*) ()

#20 0x00a49164 in boost::detail::thread_data, 
impala::Promise*), boost::_bi::list4, 
boost::_bi::va  lue, boost::_bi::value >, 
boost::_bi::value*> > > >::run()

    ()

#21 0x00c49be3 in ?? ()

#22 0x7f2058006182 in start_thread (arg=0x7f1f3023c700) at 
pthread_create.c:312

#23 0x7f20562f047d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:111

On Wed, Feb 24, 2016 at 4:44 PM, Nick Wolf  wrote:

KUDU Version 0.6.0

Two nodes of my 6 node cluster crashed with attached thread dump. This behavior 
is random and not sure what is causing the crash. 

Has anyone experienced this type of error?

Re: Spark on Kudu

2016-03-01 Thread Jean-Daniel Cryans 
Hi Ben,

AFAIK no one in the dev community committed to any timeline. I know of one
person on the Kudu Slack who's working on a better RDD, but that's about it.

Regards,

J-D

On Tue, Mar 1, 2016 at 11:00 AM, Benjamin Kim  wrote:

> Hi J-D,
>
> Quick question… Is there an ETA for KUDU-1214? I want to target a version
> of Kudu to begin real testing of Spark against it for our devs. At least, I
> can tell them what timeframe to anticipate.
>
> Just curious,
> *Benjamin Kim*
> *Data Solutions Architect*
>
> [a•mo•bee] *(n.)* the company defining digital marketing.
>
> *Mobile: +1 818 635 2900 <%2B1%20818%20635%202900>*
> 3250 Ocean Park Blvd, Suite 200  |  Santa Monica, CA 90405  |
> www.amobee.com
>
> On Feb 24, 2016, at 3:51 PM, Jean-Daniel Cryans 
> wrote:
>
> The DStream stuff isn't there at all. I'm not sure if it's needed either.
>
> The kuduRDD is just leveraging the MR input format, ideally we'd use scans
> directly.
>
> The SparkSQL stuff is there but it doesn't do any sort of pushdown. It's
> really basic.
>
> The goal was to provide something for others to contribute to. We have
> some basic unit tests that others can easily extend. None of us on the team
> are Spark experts, but we'd be really happy to assist one improve the
> kudu-spark code.
>
> J-D
>
> On Wed, Feb 24, 2016 at 3:41 PM, Benjamin Kim  wrote:
>
>> J-D,
>>
>> It looks like it fulfills most of the basic requirements (kudu RDD, kudu
>> DStream) in KUDU-1214. Am I right? Besides shoring up more Spark SQL
>> functionality (Dataframes) and doing the documentation, what more needs to
>> be done? Optimizations?
>>
>> I believe that it’s a good place to start using Spark with Kudu and
>> compare it to HBase with Spark (not clean).
>>
>> Thanks,
>> Ben
>>
>>
>> On Feb 24, 2016, at 3:10 PM, Jean-Daniel Cryans 
>> wrote:
>>
>> AFAIK no one is working on it, but we did manage to get this in for
>> 0.7.0: https://issues.cloudera.org/browse/KUDU-1321
>>
>> It's a really simple wrapper, and yes you can use SparkSQL on Kudu, but
>> it will require a lot more work to make it fast/useful.
>>
>> Hope this helps,
>>
>> J-D
>>
>> On Wed, Feb 24, 2016 at 3:08 PM, Benjamin Kim  wrote:
>>
>>> I see this KUDU-1214  targeted
>>> for 0.8.0, but I see no progress on it. When this is complete, will this
>>> mean that Spark will be able to work with Kudu both programmatically and as
>>> a client via Spark SQL? Or is there more work that needs to be done on the
>>> Spark side for it to work?
>>>
>>> Just curious.
>>>
>>> Cheers,
>>> Ben
>>>
>>>
>>
>>
>
>

Re: Kudu 0.7.0

2016-03-01 Thread Benjamin Kim 
Is there a special version of Impala coming out too?

Thanks,
Ben

> On Mar 1, 2016, at 9:51 AM, Jean-Daniel Cryans  wrote:
> 
> It will be available very soon. The thing is we (Cloudera) can't start the 
> binaries release process until the source release has been voted on.
> 
> J-D
> 
> On Tue, Mar 1, 2016 at 9:42 AM, Benjamin Kim  > wrote:
> Is the CSD for Cloudera Manager available for Kudu 0.7.0 or can we just add 
> the URL to the parcel list?
> 
> Thanks,
> Ben
> 
>

Re: Kudu 0.7.0

2016-03-01 Thread Jean-Daniel Cryans 
Yup, hold on! :)

On Tue, Mar 1, 2016 at 1:21 PM, Benjamin Kim  wrote:

> Is there a special version of Impala coming out too?
>
> Thanks,
> Ben
>
>
> On Mar 1, 2016, at 9:51 AM, Jean-Daniel Cryans 
> wrote:
>
> It will be available very soon. The thing is we (Cloudera) can't start the
> binaries release process until the source release has been voted on.
>
> J-D
>
> On Tue, Mar 1, 2016 at 9:42 AM, Benjamin Kim  wrote:
>
>> Is the CSD for Cloudera Manager available for Kudu 0.7.0 or can we just
>> add the URL to the parcel list?
>>
>> Thanks,
>> Ben
>>
>>
>
>

CISCO AND AVAYA IP Phones

2016-03-01 Thread Laison Computech Inc 
Hi,

Clean tested working pulls CPUs and QTYs in stock.

115 X X5650
65 X E5410
75 X X5660
145 X E5530
100 X E5645
40 X X5680
75 X X5690

Brand new sealed IP phones and QTYs in stock.

55 x CP-7937G
77 x CP-7942G
54 x CP-7945G
75 x CP-7962G
..
45 x Avaya 9630
65 x Avaya 9641
55 x Avaya 9640 

USED IT HARDWARE FROM:

FUJITSU   IBM SUNHP
QUANTUM   DELL   HDSSTK
NETAPP  SGI Oracle  EMC²

3Com, ADVA, Alcatel, Brocade, Cisco,
Cabletron, Enterasys, Extreme Networks,
Huawei, Marconi, Nortel, Qlogic, Avaya

Let me know if you're interested. We are very open to offers and willing to 
work with you to make sure that we have a deal.

Sincerely
Barbara Johnson
Laison Computech Inc
Tel: +1-657-205-7860
Fax: +1-347-214-0478
Email: sa...@laiisoncomputech.com
Web: www.laisoncomputech.com

CISCO AND AVAYA IP Phones

2016-03-01 Thread Laison Computech Inc 
Hi,

Clean tested working pulls CPUs and QTYs in stock.

115 X X5650
65 X E5410
75 X X5660
145 X E5530
100 X E5645
40 X X5680
75 X X5690

Brand new sealed IP phones and QTYs in stock.

55 x CP-7937G
77 x CP-7942G
54 x CP-7945G
75 x CP-7962G
..
45 x Avaya 9630
65 x Avaya 9641
55 x Avaya 9640 

USED IT HARDWARE FROM:

FUJITSU   IBM SUNHP
QUANTUM   DELL   HDSSTK
NETAPP  SGI Oracle  EMC²

3Com, ADVA, Alcatel, Brocade, Cisco,
Cabletron, Enterasys, Extreme Networks,
Huawei, Marconi, Nortel, Qlogic, Avaya

Let me know if you're interested. We are very open to offers and willing to 
work with you to make sure that we have a deal.

Sincerely
Barbara Johnson
Laison Computech Inc
Tel: +1-657-205-7860
Fax: +1-347-214-0478
Email: sa...@laiisoncomputech.com
Web: www.laisoncomputech.com