[jira] [Commented] (HIVE-10983) LazySimpleSerDe bug ,when Text is reused
[
https://issues.apache.org/jira/browse/HIVE-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581728#comment-14581728
]
Hive QA commented on HIVE-10983:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12739021/HIVE-10983.1.patch.txt
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4247/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4247/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4247/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/shims/aggregator/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/shims/aggregator/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/shims/aggregator/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-github-source-source/shims/aggregator/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-shims ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/shims/aggregator/target/hive-shims-2.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-shims ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/shims/aggregator/target/hive-shims-2.0.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-shims/2.0.0-SNAPSHOT/hive-shims-2.0.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/shims/aggregator/pom.xml
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-shims/2.0.0-SNAPSHOT/hive-shims-2.0.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Common 2.0.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-common ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/common/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/common
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-common ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-version-annotation) @
hive-common ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-common
---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/common/src/gen added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-common ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-common ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-common ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-common ---
[INFO] Compiling 74 source files to
/data/hive-ptest/working/apache-github-source-source/common/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/common/src/java/org/apache/hadoop/hive/common/JvmPauseMonitor.java:
/data/hive-ptest/working/apache-github-source-source/common/src/java/org/apache/hadoop/hive/common/JvmPauseMonitor.java
uses or overrides a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/common/src/java/org/apache/hadoop/hive/common/JvmPauseMonitor.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/common/src/java/org/apache/hadoop/hive/common/ObjectPair.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/common/src/java/org/apache/hadoop/hive/common/ObjectPair.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] ---
[jira] [Commented] (HIVE-10983) LazySimpleSerDe bug ,when Text is reused
[
https://issues.apache.org/jira/browse/HIVE-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582261#comment-14582261
]
Hive QA commented on HIVE-10983:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12739041/HIVE-10983.2.patch.txt
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9007 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join28
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4250/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4250/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4250/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12739041 - PreCommit-HIVE-TRUNK-Build
LazySimpleSerDe bug ,when Text is reused
--
Key: HIVE-10983
URL: https://issues.apache.org/jira/browse/HIVE-10983
Project: Hive
Issue Type: Bug
Components: API, CLI
Affects Versions: 0.14.0
Environment: Hadoop 2.3.0-cdh5.0.0
Hive 0.14
Reporter: xiaowei wang
Assignee: xiaowei wang
Priority: Critical
Labels: patch
Fix For: 0.14.1
Attachments: HIVE-10983.1.patch.txt, HIVE-10983.2.patch.txt
When i query data from a lzo table ， I found in results ： the length of the
current row is always largr than the previous row， and sometimes，the current
row contains the contents of the previous row。 For example ，i execute a sql
,select * from web_searchhub where logdate=2015061003, the result of sql
see blow.Notice that ,the second row content contains the first row content.
INFO [03:00:05.589] HttpFrontServer::FrontSH
msgRecv:Remote=/10.13.193.68:42098,session=3151,thread=254 2015061003
INFO [03:00:05.594] 18941e66-9962-44ad-81bc-3519f47ba274
session=901,thread=223ession=3151,thread=254 2015061003
The content of origin lzo file content see below ,just 2 rows.
INFO [03:00:05.635] b88e0473-7530-494c-82d8-e2d2ebd2666c_forweb
session=3148,thread=285
INFO [03:00:05.635] HttpFrontServer::FrontSH
msgRecv:Remote=/10.13.193.68:42095,session=3148,thread=285
I think this error is caused by the Text reuse,and I found the solutions .
Addicational, table create sql is :
CREATE EXTERNAL TABLE `web_searchhub`(
`line` string)
PARTITIONED BY (
`logdate` string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\\U'
WITH SERDEPROPERTIES (
'serialization.encoding'='GBK')
STORED AS INPUTFORMAT com.hadoop.mapred.DeprecatedLzoTextInputFormat
OUTPUTFORMAT
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat;
LOCATION
'viewfs://nsX/user/hive/warehouse/raw.db/web/web_searchhub' ；
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10983) LazySimpleSerDe bug ,when Text is reused
[ https://issues.apache.org/jira/browse/HIVE-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581882#comment-14581882 ] xiaowei wang commented on HIVE-10983: - in the first patch，i invoke a method of Text ，copyBytes().This method is added behind Hadoop1.0 ,so the compile failed. then i put up the second patch . LazySimpleSerDe bug ,when Text is reused -- Key: HIVE-10983 URL: https://issues.apache.org/jira/browse/HIVE-10983 Project: Hive Issue Type: Bug Components: API, CLI Affects Versions: 0.14.0 Environment: Hadoop 2.3.0-cdh5.0.0 Hive 0.14 Reporter: xiaowei wang Assignee: xiaowei wang Priority: Critical Labels: patch Fix For: 0.14.1 Attachments: HIVE-10983.1.patch.txt, HIVE-10983.2.patch.txt When i query data from a lzo table ， I found in results ： the length of the current row is always largr than the previous row， and sometimes，the current row contains the contents of the previous row。 For example ，i execute a sql ,select * from web_searchhub where logdate=2015061003, the result of sql see blow.Notice that ,the second row content contains the first row content. INFO [03:00:05.589] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42098,session=3151,thread=254 2015061003 INFO [03:00:05.594] 18941e66-9962-44ad-81bc-3519f47ba274 session=901,thread=223ession=3151,thread=254 2015061003 The content of origin lzo file content see below ,just 2 rows. INFO [03:00:05.635] b88e0473-7530-494c-82d8-e2d2ebd2666c_forweb session=3148,thread=285 INFO [03:00:05.635] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42095,session=3148,thread=285 I think this error is caused by the Text reuse,and I found the solutions . Addicational, table create sql is : CREATE EXTERNAL TABLE `web_searchhub`( `line` string) PARTITIONED BY ( `logdate` string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\U' WITH SERDEPROPERTIES ( 'serialization.encoding'='GBK') STORED AS INPUTFORMAT com.hadoop.mapred.DeprecatedLzoTextInputFormat OUTPUTFORMAT org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat; LOCATION 'viewfs://nsX/user/hive/warehouse/raw.db/web/web_searchhub' ； -- This message was sent by Atlassian JIRA (v6.3.4#6332)
