> wrote:
>
>> Hi Cham,
>> I'm guessing this means I won't be able to use Snowflake IO with python
>> on dataflow until then?
>> Is there a timeline for that work to be completed?
>>
>> Thank you so much for your help
>>
>>
>>
>> On F
We are working on this. ETA is the end of this quarter. Created
https://issues.apache.org/jira/browse/BEAM-11332 for tracking.
Thanks,
Cham
On Mon, Nov 23, 2020 at 5:42 AM Alan Krumholz
wrote:
> Hi Cham,
> I'm guessing this means I won't be able to use Snowflake IO with python on
>
Hi Cham,
I'm guessing this means I won't be able to use Snowflake IO with python on
dataflow until then?
Is there a timeline for that work to be completed?
Thank you so much for your help
On Fri, Nov 20, 2020 at 9:33 PM Chamikara Jayalath
wrote:
> This is because Python does not underst
This is because Python does not understand that Java specific
"beam:transform:write_files:v1" transform. Hopefully this is one of those
issues that will get resolved when we update Dataflow to directly consume
portable protos (we are working on this now).
Thanks,
Cham
On Fri, Nov 20, 2020 at
+Chamikara Jayalath any idea why this is still doing
a runner api roundtrip and failing? It's a multi-language pipeline, and
Alan has it configured to run on Dataflow runner V2.
On Fri, Nov 20, 2020 at 10:36 AM Alan Krumholz
wrote:
> Just tried that and still getting this:
>
>
Just tried that and still getting this:
---KeyError
Traceback (most recent call
last) in > 1
bq_to_snowflake( 2 'ml-betterup.coach_search.distances',
3
Hm try passing in the args as they would appear in
`sys.argv`, PipelineOptions(['--experiments=use_runner_v2'])
On Thu, Nov 19, 2020 at 12:14 PM Alan Krumholz
wrote:
> How can I pass that flag using the SDK?
> Tried this:
>
> pipeline = beam.Pipeline(options=PipelineOptions(experiments=
>>
How can I pass that flag using the SDK?
Tried this:
pipeline = beam.Pipeline(options=PipelineOptions(experiments=
> ['use_runner_v2'], ...)
but still getting a similar error:
---KeyError
DataFlow runner
On Thu, Nov 19, 2020 at 2:00 PM Brian Hulette wrote:
> Hm what runner are you using? It looks like we're trying to encode and
> decode the pipeline proto, which isn't possible for a multi-language
> pipeline. Are you using a portable runner?
>
> Brian
>
> On Thu, Nov 19, 2020 at
Hm what runner are you using? It looks like we're trying to encode and
decode the pipeline proto, which isn't possible for a multi-language
pipeline. Are you using a portable runner?
Brian
On Thu, Nov 19, 2020 at 10:42 AM Alan Krumholz
wrote:
> got it, thanks!
> I was using:
>
got it, thanks!
I was using:
'xx.us-east-1'
Seems using this instead fixes that problem:
'xx.us-east-1.snowflakecomputing.com
I'm now hitting a different error though (now in python):
in bq_to_snowflake(bq_table, snow_table,
> git_branch)
> 161 )
> 162
> --> 163 result = pipeline.run()
Hi Alan,
Sorry this error message is so verbose. What are you passing for the
server_name argument [1]? It looks like that's what the Java stacktrace is
complaining about:
java.lang.IllegalArgumentException: serverName must be in format
.snowflakecomputing.com
[1]
I'm trying to replace my custom/problematic snowflake sink with the new:
https://beam.apache.org/documentation/io/built-in/snowflake/#writing-to-snowflake
However when I try to run my pipeline (using python) I get this Java error:
RuntimeError: java.lang.RuntimeException: Failed to build
13 matches
Mail list logo
