[
https://issues.apache.org/jira/browse/BEAM-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li resolved BEAM-3247.
--
Resolution: Fixed
Fix Version/s: 2.2.0
> Sample.any memory constraint
>
[
https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596565#comment-16596565
]
Neville Li commented on BEAM-5036:
--
Yeah that's why I figured. So there's no way to reduce this overhead
[
https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596565#comment-16596565
]
Neville Li edited comment on BEAM-5036 at 8/29/18 4:23 PM:
---
Yeah that's what I
[
https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596513#comment-16596513
]
Neville Li commented on BEAM-5036:
--
{{copy+delete}} is still expensive on GCS, especially when running
[
https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596410#comment-16596410
]
Neville Li commented on BEAM-5036:
--
Yeah that's my main concern. We use GCS almost exclusively so all our
[
https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596392#comment-16596392
]
Neville Li commented on BEAM-5036:
--
If I understand this correctly, this issue affects all file based
[
https://issues.apache.org/jira/browse/BEAM-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275383#comment-16275383
]
Neville Li commented on BEAM-3234:
--
Affects 2.2.0 as well.
[
https://issues.apache.org/jira/browse/BEAM-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-3234:
-
Affects Version/s: 2.2.0
> PubsubIO batch size should be configurable
>
[
https://issues.apache.org/jira/browse/BEAM-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-991:
Fix Version/s: 2.1.0
> DatastoreIO Write should flush early for large batches
>
Neville Li created BEAM-3247:
Summary: Sample.any memory constraint
Key: BEAM-3247
URL: https://issues.apache.org/jira/browse/BEAM-3247
Project: Beam
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/BEAM-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li reassigned BEAM-3247:
Assignee: Neville Li (was: Kenneth Knowles)
> Sample.any memory constraint
>
[
https://issues.apache.org/jira/browse/BEAM-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-3234:
-
Description:
Looks like there's a payload size limit in Pubsub, and PubsubIO has a hard
coded batch size
[
https://issues.apache.org/jira/browse/BEAM-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-3234:
-
Description:
Looks like there's a payload size limit in Pubsub, and PubsubIO has a hard
coded batch size
[
https://issues.apache.org/jira/browse/BEAM-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-3234:
-
Description:
Looks like there's a payload size limit in Pubsub, and PubsubIO has a hard
coded batch size
Neville Li created BEAM-3234:
Summary: PubsubIO batch size should be configurable
Key: BEAM-3234
URL: https://issues.apache.org/jira/browse/BEAM-3234
Project: Beam
Issue Type: Bug
[
https://issues.apache.org/jira/browse/BEAM-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li reassigned BEAM-2960:
Assignee: Neville Li (was: Kenneth Knowles)
> Missing type parameter in some AvroIO.Write API
>
Neville Li created BEAM-2766:
Summary: HadoopInputFormatIO should support Void/null key/values
Key: BEAM-2766
URL: https://issues.apache.org/jira/browse/BEAM-2766
Project: Beam
Issue Type: Bug
Neville Li created BEAM-2765:
Summary: HadoopInputFormatIO should support custom key/value coder
Key: BEAM-2765
URL: https://issues.apache.org/jira/browse/BEAM-2765
Project: Beam
Issue Type:
[
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097699#comment-16097699
]
Neville Li commented on BEAM-2658:
--
However I'd still argue that {{DefaultCoder}} and
[
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097693#comment-16097693
]
Neville Li commented on BEAM-2658:
--
Types covered by each {{CoderProvider}} may overlap and we might want
[
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-2658:
-
Description:
{code}
import com.google.protobuf.Timestamp;
import org.apache.beam.sdk.Pipeline;
import
[
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-2658:
-
Description:
{code)
public class CoderTest {
public static void main(String[] args) throws
[
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-2658:
-
Description:
{code}
import com.google.protobuf.Timestamp;
import org.apache.beam.sdk.Pipeline;
import
[
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-2658:
-
Description:
{code}
public class CoderTest {
public static void main(String[] args) throws
Neville Li created BEAM-2658:
Summary: SerializableCoder has high precedence over ProtoCoder in
CoderRegistry#getCoder
Key: BEAM-2658
URL: https://issues.apache.org/jira/browse/BEAM-2658
Project: Beam
[
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-2658:
-
Summary: SerializableCoder has higher precedence over ProtoCoder in
CoderRegistry#getCoder (was:
[
https://issues.apache.org/jira/browse/BEAM-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093359#comment-16093359
]
Neville Li commented on BEAM-2453:
--
Here's an example of incorrect use of {{Combine.perKey}} that could be
[
https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090599#comment-16090599
]
Neville Li commented on BEAM-2532:
--
Would love to see a fix in the next release. This is a big performance
[
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996831#comment-15996831
]
Neville Li commented on BEAM-302:
-
Yes that ecosystem has too many build params, scala version, spark
[
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995075#comment-15995075
]
Neville Li commented on BEAM-302:
-
Looks like Spark runner still depends on 1.6.3. Can you give Spark 1.6 a
[
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994005#comment-15994005
]
Neville Li commented on BEAM-302:
-
You need the spark runner dependency which is not included by default.
>
[
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953584#comment-15953584
]
Neville Li commented on BEAM-302:
-
We prefer to keep it separate for now mainly for logistics reasons:
- we
[
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li reassigned BEAM-302:
---
Assignee: (was: Neville Li)
> Add Scio Scala DSL to Beam
> --
>
>
[
https://issues.apache.org/jira/browse/BEAM-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li closed BEAM-1518.
Resolution: Fixed
> Support deflate (zlib) in CompressedSource and FileBasedSink
>
[
https://issues.apache.org/jira/browse/BEAM-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-1518:
-
Fix Version/s: 0.6.0
> Support deflate (zlib) in CompressedSource and FileBasedSink
>
Neville Li created BEAM-1520:
Summary: Implement TFRecordIO (Reading/writing Tensorflow Standard
format)
Key: BEAM-1520
URL: https://issues.apache.org/jira/browse/BEAM-1520
Project: Beam
Issue
[
https://issues.apache.org/jira/browse/BEAM-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li reassigned BEAM-1520:
Assignee: Neville Li (was: Davor Bonaci)
> Implement TFRecordIO (Reading/writing Tensorflow
[
https://issues.apache.org/jira/browse/BEAM-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li reassigned BEAM-1519:
Assignee: (was: Neville Li)
> Support snappy in CompressedSource and FileBasedSink
>
[
https://issues.apache.org/jira/browse/BEAM-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-1518:
-
Summary: Support deflate (zlib) in CompressedSource and FileBasedSink
(was: Support ZLIB (deflate) in
[
https://issues.apache.org/jira/browse/BEAM-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-1518:
-
Description: `.deflate` files are quite common in Hadoop and also supported
by TensorFlow in TFRecord file
Neville Li created BEAM-1519:
Summary: CLONE - Support snappy in CompressedSource and
FileBasedSink
Key: BEAM-1519
URL: https://issues.apache.org/jira/browse/BEAM-1519
Project: Beam
Issue Type:
[
https://issues.apache.org/jira/browse/BEAM-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Li updated BEAM-1519:
-
Summary: Support snappy in CompressedSource and FileBasedSink (was: CLONE
- Support snappy in
Neville Li created BEAM-1518:
Summary: Support ZLIB (deflate) in CompressedSource and
FileBasedSink
Key: BEAM-1518
URL: https://issues.apache.org/jira/browse/BEAM-1518
Project: Beam
Issue Type:
[
https://issues.apache.org/jira/browse/BEAM-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862201#comment-15862201
]
Neville Li edited comment on BEAM-298 at 2/11/17 4:12 AM:
--
That didn't work for me.
[
https://issues.apache.org/jira/browse/BEAM-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862201#comment-15862201
]
Neville Li commented on BEAM-298:
-
That didn't work for me. I had to add it as a {compile} scope.
> Make
[
https://issues.apache.org/jira/browse/BEAM-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861795#comment-15861795
]
Neville Li edited comment on BEAM-298 at 2/10/17 8:08 PM:
--
As a result of this
[
https://issues.apache.org/jira/browse/BEAM-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861795#comment-15861795
]
Neville Li commented on BEAM-298:
-
As a result of this change I need to include {{junit}} in my dependencies
[
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836553#comment-15836553
]
Neville Li commented on BEAM-302:
-
WIP branch here using 0.4.0
48 matches
Mail list logo
