[ https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283763#comment-15283763 ]
Hive QA commented on HIVE-13622: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12803981/HIVE-13622.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/288/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/288/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-288/ Messages: {noformat} **** This message was trimmed, see log for full details **** [INFO] Excluding org.apache.spark:spark-core_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding com.twitter:chill_2.10:jar:0.5.0 from the shaded jar. [INFO] Excluding com.twitter:chill-java:jar:0.5.0 from the shaded jar. [INFO] Excluding org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-launcher_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-network-common_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-network-shuffle_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-unsafe_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from the shaded jar. [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 from the shaded jar. [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from the shaded jar. [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the shaded jar. [INFO] Excluding com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded jar. [INFO] Excluding com.typesafe.akka:akka-actor_2.10:jar:2.3.11 from the shaded jar. [INFO] Excluding com.typesafe:config:jar:1.2.1 from the shaded jar. [INFO] Excluding org.uncommons.maths:uncommons-maths:jar:1.2.2a from the shaded jar. [INFO] Excluding com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11 from the shaded jar. [INFO] Excluding org.scala-lang:scala-library:jar:2.10.4 from the shaded jar. [INFO] Excluding org.json4s:json4s-jackson_2.10:jar:3.2.10 from the shaded jar. [INFO] Excluding org.json4s:json4s-core_2.10:jar:3.2.10 from the shaded jar. [INFO] Excluding org.json4s:json4s-ast_2.10:jar:3.2.10 from the shaded jar. [INFO] Excluding org.scala-lang:scalap:jar:2.10.0 from the shaded jar. [INFO] Excluding org.scala-lang:scala-compiler:jar:2.10.0 from the shaded jar. [INFO] Excluding org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1 from the shaded jar. [INFO] Excluding com.clearspring.analytics:stream:jar:2.7.0 from the shaded jar. [INFO] Excluding io.dropwizard.metrics:metrics-graphite:jar:3.1.2 from the shaded jar. [INFO] Excluding com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.4 from the shaded jar. [INFO] Excluding org.scala-lang:scala-reflect:jar:2.10.4 from the shaded jar. [INFO] Excluding oro:oro:jar:2.0.8 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-client:jar:0.8.2 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-underfs-hdfs:jar:0.8.2 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-underfs-s3:jar:0.8.2 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-underfs-local:jar:0.8.2 from the shaded jar. [INFO] Excluding net.razorvine:pyrolite:jar:4.9 from the shaded jar. [INFO] Excluding net.sf.py4j:py4j:jar:0.9 from the shaded jar. [INFO] Excluding org.spark-project.spark:unused:jar:1.0.0 from the shaded jar. [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.10 from the shaded jar. [INFO] Replacing original artifact with shaded artifact. [INFO] Replacing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT.jar with /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT-shaded.jar [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-exec --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT.pom [INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT-tests.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT-tests.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT-core.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT-core.jar [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Llap Server 2.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-llap-server --- [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/llap-server/target [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/llap-server (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-llap-server --- [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-llap-server --- [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/llap-server/src/gen/protobuf/gen-java added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/llap-server/src/gen/thrift/gen-javabean added. [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-llap-server --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-llap-server --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 19 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-llap-server --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-llap-server --- [INFO] Compiling 87 source files to /data/hive-ptest/working/apache-github-source-source/llap-server/target/classes [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[29,16] sun.misc.Cleaner is internal proprietary API and may be removed in a future release [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[29,16] sun.misc.Cleaner is internal proprietary API and may be removed in a future release [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[29,16] sun.misc.Cleaner is internal proprietary API and may be removed in a future release [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[74,9] sun.misc.Cleaner is internal proprietary API and may be removed in a future release [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java: Some input files use or override a deprecated API. [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java: Recompile with -Xlint:deprecation for details. [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/DirWatcher.java: Some input files use unchecked or unsafe operations. [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/DirWatcher.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-llap-server --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-llap-server --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/llap-server/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/llap-server/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/llap-server/target/tmp/conf [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/llap-server/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-llap-server --- [INFO] Compiling 13 source files to /data/hive-ptest/working/apache-github-source-source/llap-server/target/test-classes [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java: Some input files use unchecked or unsafe operations. [WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-llap-server --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-llap-server --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-llap-server --- [INFO] [INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-llap-server --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT-tests.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-llap-server --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-llap-server/2.1.0-SNAPSHOT/hive-llap-server-2.1.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/llap-server/pom.xml to /home/hiveptest/.m2/repository/org/apache/hive/hive-llap-server/2.1.0-SNAPSHOT/hive-llap-server-2.1.0-SNAPSHOT.pom [INFO] Installing /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT-tests.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-llap-server/2.1.0-SNAPSHOT/hive-llap-server-2.1.0-SNAPSHOT-tests.jar [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Service 2.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ Downloading: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml [WARNING] Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to apache.snapshots (http://repository.apache.org/snapshots): Failed to transfer file: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml. Return code is: 503 , ReasonPhrase:Service Unavailable. [WARNING] Failure to transfer org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced. Original error: Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to apache.snapshots (http://repository.apache.org/snapshots): Failed to transfer file: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml. Return code is: 503 , ReasonPhrase:Service Unavailable. Downloading: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [3.148s] [INFO] Hive Shims Common ................................. SUCCESS [5.074s] [INFO] Hive Shims 0.23 ................................... SUCCESS [3.304s] [INFO] Hive Shims Scheduler .............................. SUCCESS [0.908s] [INFO] Hive Shims ........................................ SUCCESS [0.773s] [INFO] Hive Storage API .................................. SUCCESS [1.708s] [INFO] Hive ORC .......................................... SUCCESS [2.892s] [INFO] Hive Common ....................................... SUCCESS [4.510s] [INFO] Hive Service RPC .................................. SUCCESS [2.754s] [INFO] Hive Serde ........................................ SUCCESS [3.765s] [INFO] Hive Metastore .................................... SUCCESS [17.064s] [INFO] Hive Ant Utilities ................................ SUCCESS [0.503s] [INFO] Hive Llap Common .................................. SUCCESS [4.747s] [INFO] Hive Llap Client .................................. SUCCESS [1.150s] [INFO] Hive Llap Tez ..................................... SUCCESS [1.257s] [INFO] Spark Remote Client ............................... SUCCESS [2.344s] [INFO] Hive Query Language ............................... SUCCESS [48.559s] [INFO] Hive Llap Server .................................. SUCCESS [2.685s] [INFO] Hive Service ...................................... FAILURE [11.141s] [INFO] Hive Accumulo Handler ............................. SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HPL/SQL ...................................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive Llap External Client ......................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:59.396s [INFO] Finished at: Sun May 15 01:59:26 GMT 2016 [INFO] Final Memory: 157M/1179M [INFO] ------------------------------------------------------------------------ [WARNING] The requested profile "hadoop-1" could not be activated because it does not exist. [ERROR] Failed to execute goal on project hive-service: Could not resolve dependencies for project org.apache.hive:hive-service:jar:2.1.0-SNAPSHOT: Failed to collect dependencies for [org.apache.hive:hive-exec:jar:2.1.0-SNAPSHOT (compile), org.apache.hive:hive-metastore:jar:2.1.0-SNAPSHOT (compile), org.apache.hive:hive-service-rpc:jar:2.1.0-SNAPSHOT (compile), org.apache.hive:hive-llap-server:jar:2.1.0-SNAPSHOT (compile), commons-codec:commons-codec:jar:1.4 (compile), commons-cli:commons-cli:jar:1.2 (compile), net.sf.jpam:jpam:jar:1.1 (compile), commons-lang:commons-lang:jar:2.6 (compile), org.eclipse.jetty.aggregate:jetty-all:jar:7.6.0.v20120127 (compile), tomcat:jasper-compiler:jar:5.5.23 (compile), tomcat:jasper-runtime:jar:5.5.23 (compile), org.apache.thrift:libfb303:jar:0.9.3 (compile), org.apache.thrift:libthrift:jar:0.9.3 (compile), org.apache.curator:curator-framework:jar:2.6.0 (compile), org.apache.curator:curator-recipes:jar:2.6.0 (compile), org.apache.hadoop:hadoop-common:jar:2.6.0 (compile?), org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0 (compile?), org.jamon:jamon-runtime:jar:2.3.1 (compile), org.apache.hive:hive-exec:jar:tests:2.1.0-SNAPSHOT (test), org.apache.hive:hive-common:jar:tests:2.1.0-SNAPSHOT (test), junit:junit:jar:4.11 (test), org.apache.directory.client.ldap:ldap-client-api:jar:0.1 (test), org.apache.directory.server:apacheds-server-integ:jar:1.5.6 (test), org.apache.directory.server:apacheds-test-framework:jar:1.5.6 (test), org.slf4j:slf4j-api:jar:1.7.10 (compile)]: Failed to read artifact descriptor for org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: Could not transfer artifact org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT from/to apache.snapshots (http://repository.apache.org/snapshots): Failed to transfer file: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom. Return code is: 503 , ReasonPhrase:Service Unavailable. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-service + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12803981 - PreCommit-HIVE-MASTER-Build > WriteSet tracking optimizations > ------------------------------- > > Key: HIVE-13622 > URL: https://issues.apache.org/jira/browse/HIVE-13622 > Project: Hive > Issue Type: Bug > Components: Transactions > Affects Versions: 1.3.0, 2.1.0 > Reporter: Eugene Koifman > Assignee: Eugene Koifman > Priority: Critical > Attachments: HIVE-13622.2.patch > > > HIVE-13395 solves the the lost update problem with some inefficiencies. > 1. TxhHandler.OperationType is currently derived from LockType. This doesn't > distinguish between Update and Delete but would be useful. See comments in > TxnHandler. Should be able to pass in Insert/Update/Delete info from client > into TxnHandler. > 2. TxnHandler.addDynamicPartitions() should know the OperationType as well > from the client. It currently extrapolates it from TXN_COMPONENTS. This > works but requires extra SQL statements and is thus less performant. It will > not work multi-stmt txns. See comments in the code. > 3. TxnHandler.checkLock() see more comments around > "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being > called as part of an op running with dynamic partitions, it could be more > efficient. In that case we don't have to write to TXN_COMPONENTS at all > during lock acquisition. Conversely, if not running with DynPart then, we > can kill current txn on lock grant rather than wait until commit time. > 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing > multiple rows into single SQL stmt (but with a limit for extreme cases) > 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that > are only being read to TXN_COMPONENTS. This is useless at best since read > op don't generate anything to compact. For example, delete from T where t1 > in (select c1 from C) - no reason to add C to txn_components but we do. > > All of these require some Thrift changes > Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11() > Also see comments in > [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712] -- This message was sent by Atlassian JIRA (v6.3.4#6332)