[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (37 issues) Subscriber: pigdaily Key Summary PIG-5236json simple jar not included automatically while trying to load multiple schema in pig using avro https://issues.apache.org/jira/browse/PIG-5236 PIG-5225Several unit tests are not annotated with @Test https://issues.apache.org/jira/browse/PIG-5225 PIG-5207BugFix e2e tests fail on spark https://issues.apache.org/jira/browse/PIG-5207 PIG-5194HiveUDF fails with Spark exec type https://issues.apache.org/jira/browse/PIG-5194 PIG-5185Job name show "DefaultJobName" when running a Python script https://issues.apache.org/jira/browse/PIG-5185 PIG-5184set command to view value of a variable https://issues.apache.org/jira/browse/PIG-5184 PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown NPE in multithread env https://issues.apache.org/jira/browse/PIG-5160 PIG-5115Builtin AvroStorage generates incorrect avro schema when the same pig field name appears in the alias https://issues.apache.org/jira/browse/PIG-5115 PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive set to true https://issues.apache.org/jira/browse/PIG-5106 PIG-5081Can not run pig on spark source code distribution https://issues.apache.org/jira/browse/PIG-5081 PIG-5080Support store alias as spark table https://issues.apache.org/jira/browse/PIG-5080 PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput https://issues.apache.org/jira/browse/PIG-5057 PIG-5029Optimize sort case when data is skewed https://issues.apache.org/jira/browse/PIG-5029 PIG-4926Modify the content of start.xml for spark mode https://issues.apache.org/jira/browse/PIG-4926 PIG-4913Reduce jython function initiation during compilation https://issues.apache.org/jira/browse/PIG-4913 PIG-4849pig on tez will cause tez-ui to crash,because the content from timeline server is too long. https://issues.apache.org/jira/browse/PIG-4849 PIG-4750REPLACE_MULTI should compile Pattern once and reuse it https://issues.apache.org/jira/browse/PIG-4750 PIG-4748DateTimeWritable forgets Chronology https://issues.apache.org/jira/browse/PIG-4748 PIG-4745DataBag should protect content of passed list of tuples https://issues.apache.org/jira/browse/PIG-4745 PIG-4684Exception should be changed to warning when job diagnostics cannot be fetched https://issues.apache.org/jira/browse/PIG-4684 PIG-4656Improve String serialization and comparator performance in BinInterSedes https://issues.apache.org/jira/browse/PIG-4656 PIG-4598Allow user defined plan optimizer rules https://issues.apache.org/jira/browse/PIG-4598 PIG-4551Partition filter is not pushed down in case of SPLIT https://issues.apache.org/jira/browse/PIG-4551 PIG-4539New PigUnit https://issues.apache.org/jira/browse/PIG-4539 PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException https://issues.apache.org/jira/browse/PIG-4515 PIG-4323PackageConverter hanging in Spark https://issues.apache.org/jira/browse/PIG-4323 PIG-4313StackOverflowError in LIMIT operation on Spark https://issues.apache.org/jira/browse/PIG-4313 PIG-4251Pig on Storm https://issues.apache.org/jira/browse/PIG-4251 PIG-4002Disable combiner when map-side aggregation is used https://issues.apache.org/jira/browse/PIG-4002 PIG-3952PigStorage accepts '-tagSplit' to return full split information https://issues.apache.org/jira/browse/PIG-3952 PIG-3911Define unique fields with @OutputSchema https://issues.apache.org/jira/browse/PIG-3911 PIG-3877Getting Geo Latitude/Longitude from Address Lines https://issues.apache.org/jira/browse/PIG-3877 PIG-3873Geo distance calculation using Haversine https://issues.apache.org/jira/browse/PIG-3873 PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange handling of Daylight Saving Time with location based timezones https://issues.apache.org/jira/browse/PIG-3864 PIG-3668COR built-in function when atleast one of the coefficient values is NaN https://issues.apache.org/jira/browse/PIG-3668 PIG-3587add functionality for rolling over dates https://issues.apache.org/jira/browse/PIG-3587 PIG-1804Alow Jython function to implement Algebraic and/or Accumulator interfaces https://issues.apache.org/jira/browse/PIG-1804 You may edit this subscription at: https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328&filterId=12322384
[jira] [Commented] (PIG-5199) exclude jline in spark dependency
[
https://issues.apache.org/jira/browse/PIG-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015965#comment-16015965
]
Adam Szita commented on PIG-5199:
-
I think it's because the {{buildJar}} macrodef needs this attribute. This
macrodef is responsible to create a jar and we use it for both the core and fat
jar generation.
In the core case we currently don't package any other lib into the jar but we
still need to tell this to the macrodef and pass an empty list.
If you were to remove the attribute from the buildJar macrodef call (e.g.
within "jar" target) you'd get an error:
{code}
BUILD FAILED
/Users/szita/shadow/CDH/pig/build.xml:691: required attribute includedjars not
set
{code}
> exclude jline in spark dependency
> -
>
> Key: PIG-5199
> URL: https://issues.apache.org/jira/browse/PIG-5199
> Project: Pig
> Issue Type: Sub-task
> Components: spark
>Reporter: liyunzhang_intel
>Assignee: Adam Szita
> Fix For: spark-branch
>
> Attachments: PIG-5199.1.patch, PIG-5199.patch
>
>
> when i fix PIG-5197 and run TestGrunt, the exception is thrown
> {code}
> [ERROR] Terminal initialization failed; falling back to unsupported$
> 4220 java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but
> interface was expected$
> 4221 ^Iat jline.TerminalFactory.create(TerminalFactory.java:101)$
> 4222 ^Iat jline.TerminalFactory.get(TerminalFactory.java:159)$
> 4223 ^Iat jline.console.ConsoleReader.(ConsoleReader.java:227)$
> 4224 ^Iat jline.console.ConsoleReader.(ConsoleReader.java:219)$
> 4225 ^Iat jline.console.ConsoleReader.(ConsoleReader.java:211)$
> 4226 ^Iat org.apache.pig.Main.run(Main.java:554)$
> 4227 ^Iat org.apache.pig.PigRunner.run(PigRunner.java:49)$
> 4228 ^Iat org.apache.pig.test.TestGrunt.testGruntUtf8(TestGrunt.java:1579)$
> 4229 ^Iat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)$
> 4230 ^Iat
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)$
> 4231 ^Iat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)$
> 4232 ^Iat java.lang.reflect.Method.invoke(Method.java:498)$
> 4233 ^Iat
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)$
> 4234 ^Iat
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)$
> 4235 ^Iat
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)$
> 4236 ^Iat
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)$
> 4237 ^Iat
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)$
> 4238 ^Iat org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)$
> 4239 ^Iat
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)$
> 4240 ^Iat
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)$
> 4241 ^Iat org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)$
> 4242 ^Iat org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)$
> 4243 ^Iat org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)$
> 4244 ^Iat org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)$
> {code}
> i found this is because there are 2 jars about jline in different version
> {code}
> find -name jline*jar
> ./build/ivy/lib/spark/jline-0.9.94.jar
> ./build/ivy/lib/Pig/jline-2.11.jar
> ./lib/spark/jline-0.9.94.jar
> ./lib/jline-2.11.jar
> {code}
> we need to exclude jline-0.9.94 from spark dependency.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PIG-5235) Typecast with as-clause fails for tuple/bag with an empty schema
[
https://issues.apache.org/jira/browse/PIG-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015951#comment-16015951
]
Nandor Kollar commented on PIG-5235:
+1
> Typecast with as-clause fails for tuple/bag with an empty schema
>
>
> Key: PIG-5235
> URL: https://issues.apache.org/jira/browse/PIG-5235
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
> Attachments: pig-5235-v01.patch
>
>
> Following script fails with trunk(0.17).
> {code}
> a = load 'test.txt' as (mytuple:tuple (), gpa:float);
> b = foreach a generate mytuple as (mytuple2:(name:int, age:double));
> store b into '/tmp/deleteme';
> {code}
> 2017-05-16 09:52:31,280 \[main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2999: Unexpected internal error. null
> (This is a continuation from the as-clause fix at PIG-2315 and follow up jira
> PIG-4933)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (PIG-5237) Fix DOT file parsing to enable DOT-based physical plan testing
[ https://issues.apache.org/jira/browse/PIG-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5237: Priority: Minor (was: Major) > Fix DOT file parsing to enable DOT-based physical plan testing > -- > > Key: PIG-5237 > URL: https://issues.apache.org/jira/browse/PIG-5237 > Project: Pig > Issue Type: Bug > Components: tools >Affects Versions: 0.16.0 >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (PIG-5237) Fix DOT file parsing to enable DOT-based physical plan testing
Adam Szita created PIG-5237: --- Summary: Fix DOT file parsing to enable DOT-based physical plan testing Key: PIG-5237 URL: https://issues.apache.org/jira/browse/PIG-5237 Project: Pig Issue Type: Bug Components: tools Affects Versions: 0.16.0 Reporter: Adam Szita Assignee: Adam Szita -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PIG-5135) HDFS bytes read stats are always 0 in Spark mode
[
https://issues.apache.org/jira/browse/PIG-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015412#comment-16015412
]
Adam Szita commented on PIG-5135:
-
[~kellyzly] Thanks for catching this - it was probably missing {{svn add}}
calls during the original commit. I can see now it's fixed by committing the
missing files
> HDFS bytes read stats are always 0 in Spark mode
>
>
> Key: PIG-5135
> URL: https://issues.apache.org/jira/browse/PIG-5135
> Project: Pig
> Issue Type: Bug
> Components: spark
>Reporter: liyunzhang_intel
>Assignee: Adam Szita
> Fix For: spark-branch
>
> Attachments: PIG-5135.0.patch, PIG-5135.1.patch, PIG-5135.2.patch
>
>
> I discovered this while running TestOrcStoragePushdown unit test in Spark
> mode where the test depends on the value of this stat.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Re: Review Request 57317: Support Pig On Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57317/ --- (Updated May 18, 2017, 8:06 a.m.) Review request for pig, Daniel Dai and Rohini Palaniswamy. Changes --- update the patch with the latest branch code(e1c22a8) Bugs: PIG-4059 and PIG-4854; https://issues.apache.org/jira/browse/PIG-4059 https://issues.apache.org/jira/browse/PIG-4854; Repository: pig-git Description --- Merge all changes from spark branch Diffs (updated) - bin/pig e1212fa build.xml a0d2ca8 ivy.xml 42daec9 ivy/libraries.properties 481066e src/META-INF/services/org.apache.pig.ExecType 5c034c8 src/docs/src/documentation/content/xdocs/start.xml c9a1491 src/org/apache/pig/PigConfiguration.java d25f81a src/org/apache/pig/PigWarning.java fcda114 src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java ac03d40 src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/NoopFilterRemover.java 4d91556 src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/NoopFilterRemoverUtil.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigHadoopLogger.java 255650e src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigInputFormat.java 6fe8ff3 src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSplit.java e866b28 src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizerMR.java 8170f02 src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java 0e35273 src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java ecf780c src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java 3bad98b src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhysicalPlan.java 2376d03 src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POBroadcastSpark.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCollectedGroup.java bcbfe2b src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POFRJoin.java d80951a src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POFRJoinSpark.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java 4dc6d54 src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POGlobalRearrange.java 52cfb73 src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMergeCogroup.java 4923d3f src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMergeJoin.java 13f70c0 src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPoissonSample.java f2830c2 src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSort.java c3a82c3 src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/KryoSerializer.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/MapReducePartitionerWrapper.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkEngineConf.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkExecType.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkExecutionEngine.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLocalExecType.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkPOUserFuncVisitor.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkPigContext.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkPigRecordReader.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkPigSplit.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/SparkUtil.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/UDFJarsFinder.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/converter/BroadcastConverter.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java PRE-CREATION src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CounterConverter.java PRE-CREATION src
