Unsubscribe
Hi: Plz Unsubscribe me. Thanks! - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: How to implement a "saveAsBinaryFile" function?
Hi Fokko, Maxim, Long:
Thanks!
This reading has been occurred in a custom datasource as below:
override def createRelation(…) {
…
blocks.map(block => (block.bytes)).saveAsTextFile(parameters("path”))
...
}
I am a new Sparker, will try the those methods you guys provides.
Best!
Bing.
On Jan 17, 2020, at 4:28 AM, Maxim Gekk
mailto:maxim.g...@databricks.com>> wrote:
Hi Bing,
You can try Text datasource. It shouldn't modify strings:
scala>
Seq(""""20192_1",1,24,0,2,”S66.000x001”""").toDS.write.text("tmp/text.txt")
$ cat tmp/text.txt/part-0-256d960f-9f85-47fe-8edd-8428276eb3c6-c000.txt
"20192_1",1,24,0,2,”S66.000x001”
Maxim Gekk
Software Engineer
Databricks B. V.
[http://go.databricks.com/hubfs/emails/Databricks-logo-bug.png]
<http://databricks.com/>
On Thu, Jan 16, 2020 at 10:02 PM Long, Andrew
mailto:loand...@amazon.com.invalid>> wrote:
Hey Bing,
There’s a couple different approaches you could take. The quickest and easiest
would be to use the existing APIs
val bytes = spark.range(1000
bytes.foreachPartition(bytes =>{
//W ARNING anything used in here will need to be serializable.
// There's some magic to serializing the hadoop conf. see the hadoop wrapper
class in the source
val writer = FileSystem.get(null).create(new Path("s3://..."))
bytes.foreach(b => writer.write(b))
writer.close()
})
The more complicated but pretty approach would be to either implement a custom
datasource.
From: "Duan,Bing" mailto:duanb...@baidu.com>>
Date: Thursday, January 16, 2020 at 12:35 AM
To: "dev@spark.apache.org<mailto:dev@spark.apache.org>"
mailto:dev@spark.apache.org>>
Subject: How to implement a "saveAsBinaryFile" function?
Hi all:
I read binary data(protobuf format) from filesystem by binaryFiles function to
a RDD[Array[Byte]] it works fine. But when I save the it to filesystem by
saveAsTextFile, the quotation mark was be escaped like this:
"\"20192_1\"",1,24,0,2,"\"S66.000x001\””,which should be
"20192_1",1,24,0,2,”S66.000x001”.
Anyone could give me some tip to implement a function like saveAsBinaryFile to
persist the RDD[Array[Byte]]?
Bests!
Bing
How to implement a "saveAsBinaryFile" function?
Hi all: I read binary data(protobuf format) from filesystem by binaryFiles function to a RDD[Array[Byte]] it works fine. But when I save the it to filesystem by saveAsTextFile, the quotation mark was be escaped like this: "\"20192_1\"",1,24,0,2,"\"S66.000x001\””,which should be "20192_1",1,24,0,2,”S66.000x001”. Anyone could give me some tip to implement a function like saveAsBinaryFile to persist the RDD[Array[Byte]]? Bests! Bing
unsubscribe
Unsubscribe. 发自我的小米手机 在 2019年9月10日 上午8:32，Matei Zaharia 写道： Hi all, The Spark PMC recently voted to add several new committers and one PMC member. Join me in welcoming them to their new roles! New PMC member: Dongjoon Hyun New committers: Ryan Blue, Liang-Chi Hsieh, Gengliang Wang, Yuming Wang, Weichen Xu, Ruifeng Zheng The new committers cover lots of important areas including ML, SQL, and data sources, so it’s great to have them here. All the best, Matei and the Spark PMC - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
unsubscribe
unsubscribe. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: unsubscribe
unsubscribe On Aug 1, 2019, at 6:36 PM, abel palaty mailto:palatya...@gmail.com>> wrote:
Re: Remain an unused import function in wasm
Hi, Sam:
Yes, I have an implementation based on WAVM, design doc :
https://github.com/duanbing/WAVM/blob/master/Gas.md .
'Sam Clegg' via emscripten-discuss
于2019年5月4日周六 下午3:14写道：
> One way to achieve this might be to use binaryen tools to inject your wasm
> code into the module. One of the simplest ways would be use wasm-dis to
> turn the module into text, when wasm-as to re-assembly it after adding your
> new imports and code.
>
> *From: *Duan Bing
> *Date: *Fri, May 3, 2019 at 8:14 AM
> *To: *
>
> Thanks, Floh.
>> I didn't compile it into a library, so visibility attribute seems not
>> take effect.
>> As we can see from emscripth.h
>> #define EMSCRIPTEN_KEEPALIVE __attribute__((used)) __attribute__
>> ((visibility ("default"))) actually, this macro affects the function
>> with definition, but not for declaration only from my trials.
>>
>>
>> Floh 于2019年5月3日周五 上午1:12写道：
>>
>>> PS: I've been looking around a bit, and maybe this is the better
>>> solution (via __attribute__(...)):
>>>
>>> __attribute__((visibility("default"))) void foo(void);
>>>
>>> This should be the clang equivalent to __declspec(dllimport) in the
>>> Visual Studio compiler, see here:
>>>
>>> https://webassembly.org/docs/dynamic-linking/
>>>
>>> On Thursday, 2 May 2019 17:06:27 UTC+2, Duan Bing wrote:
>>>>
>>>> Thanks, your new idea works for me. the motivation of doing this is
>>>> that I will call this imported function while I will insert some wasm code
>>>> to the original wasm file later, similar to this project had done
>>>> https://github.com/ewasm/wasm-metering , but by a different PL. so I
>>>> don't want the original wasm developer to use this function(I will insert
>>>> foo dynamically with a mangling name ), and I also occupy the import index
>>>> space by doing this in advance.
>>>>
>>>> Floh 于2019年5月2日周四 下午10:56写道：
>>>>
>>>>> Ah ok, I misunderstood your question, apologies. I don't know how one
>>>>> would add a function declaration to the WASM imports table like this.
>>>>> EMSCRIPTEN_KEEPALIVE and the "-s EXPORTED_FUNCTIONS" linker option would
>>>>> work for the "other side" (some WASM module exporting functions to the
>>>>> outside).
>>>>>
>>>>> Complete guess, but maybe you need some "dummy code" that's guaranteed
>>>>> to stay in the importing WASM module, even though this is never called? So
>>>>> that the linker would recognize the function as an "import".
>>>>>
>>>>> Something like this:
>>>>>
>>>>> extern "C" void foo();
>>>>> // dummy_imports() is never called, but will not be removed because of
>>>>> EMSCRIPTEN_KEEPALIVE
>>>>> EMSCRIPTEN_KEEPALIVE void dummy_imports(void) {
>>>>>foo();
>>>>> }
>>>>>
>>>>> But as I said, this is complete guess.
>>>>>
>>>>> On Thursday, 2 May 2019 16:22:28 UTC+2, Duan Bing wrote:
>>>>>>
>>>>>> Thanks, Floh. it works when the function is defined.
>>>>>> Actually, my function is an external function declaration, like
>>>>>> extern "C" void foo();
>>>>>>
>>>>>> It doesn't work even I declare like this:
>>>>>> extern "C" void EMSCRIPTEN_KEEPALIVE foo().
>>>>>>
>>>>>> Do you have any idea for this situation?
>>>>>>
>>>>>> Floh 于2019年5月2日周四 下午10:13写道：
>>>>>>
>>>>>>> Have a look at EMSCRIPTEN_KEEPALIVE:
>>>>>>> https://emscripten.org/docs/api_reference/emscripten.h.html#c.EMSCRIPTEN_KEEPALIVE,
>>>>>>> this prevents a C/C++ function from being removed even when it isn't
>>>>>>> referenced by other C/C++ code (normally this is used to call C
>>>>>>> functions
>>>>>>> from the JS side).
>>>>>>>
>>>>>>> On Thursday, 2 May 2019 08:16:43 UTC+2, Duan Bing wrote:
>>>>>>>>
>>>>>>>> I declare an external function in C source file, but never used by
>>>>>>>> the code, then compile it to wasm by EMCC. there is no doubt the
>>>>>>>> external
>>>>>>>
Re: Remain an unused import function in wasm
Thanks, your new idea works for me. the motivation of doing this is that
I will call this imported function while I will insert some wasm code to
the original wasm file later, similar to this project had done
https://github.com/ewasm/wasm-metering , but by a different PL. so I don't
want the original wasm developer to use this function(I will insert foo
dynamically with a mangling name ), and I also occupy the import index
space by doing this in advance.
Floh 于2019年5月2日周四 下午10:56写道：
> Ah ok, I misunderstood your question, apologies. I don't know how one
> would add a function declaration to the WASM imports table like this.
> EMSCRIPTEN_KEEPALIVE and the "-s EXPORTED_FUNCTIONS" linker option would
> work for the "other side" (some WASM module exporting functions to the
> outside).
>
> Complete guess, but maybe you need some "dummy code" that's guaranteed to
> stay in the importing WASM module, even though this is never called? So
> that the linker would recognize the function as an "import".
>
> Something like this:
>
> extern "C" void foo();
> // dummy_imports() is never called, but will not be removed because of
> EMSCRIPTEN_KEEPALIVE
> EMSCRIPTEN_KEEPALIVE void dummy_imports(void) {
> foo();
> }
>
> But as I said, this is complete guess.
>
> On Thursday, 2 May 2019 16:22:28 UTC+2, Duan Bing wrote:
>>
>> Thanks, Floh. it works when the function is defined.
>> Actually, my function is an external function declaration, like
>> extern "C" void foo();
>>
>> It doesn't work even I declare like this:
>> extern "C" void EMSCRIPTEN_KEEPALIVE foo().
>>
>> Do you have any idea for this situation?
>>
>> Floh 于2019年5月2日周四 下午10:13写道：
>>
>>> Have a look at EMSCRIPTEN_KEEPALIVE:
>>> https://emscripten.org/docs/api_reference/emscripten.h.html#c.EMSCRIPTEN_KEEPALIVE,
>>> this prevents a C/C++ function from being removed even when it isn't
>>> referenced by other C/C++ code (normally this is used to call C functions
>>> from the JS side).
>>>
>>> On Thursday, 2 May 2019 08:16:43 UTC+2, Duan Bing wrote:
>>>>
>>>> I declare an external function in C source file, but never used by the
>>>> code, then compile it to wasm by EMCC. there is no doubt the external
>>>> function will be eliminated by the compiler.
>>>>
>>>> Is there any approach to keep this external function and then be
>>>> compiled to an import function in wasm?
>>>>
>>>> The motivation to remain the external function is that I want to insert
>>>> some wasm code in the wasm file after it's generated.
>>>>
>>>> Thanks all!
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "emscripten-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to emscripten-discuss+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to emscripten-discuss+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Remain an unused import function in wasm
Thanks, Floh. it works when the function is defined. Actually, my function is an external function declaration, like extern "C" void foo(); It doesn't work even I declare like this: extern "C" void EMSCRIPTEN_KEEPALIVE foo(). Do you have any idea for this situation? Floh 于2019年5月2日周四 下午10:13写道： > Have a look at EMSCRIPTEN_KEEPALIVE: > https://emscripten.org/docs/api_reference/emscripten.h.html#c.EMSCRIPTEN_KEEPALIVE, > this prevents a C/C++ function from being removed even when it isn't > referenced by other C/C++ code (normally this is used to call C functions > from the JS side). > > On Thursday, 2 May 2019 08:16:43 UTC+2, Duan Bing wrote: >> >> I declare an external function in C source file, but never used by the >> code, then compile it to wasm by EMCC. there is no doubt the external >> function will be eliminated by the compiler. >> >> Is there any approach to keep this external function and then be compiled >> to an import function in wasm? >> >> The motivation to remain the external function is that I want to insert >> some wasm code in the wasm file after it's generated. >> >> Thanks all! >> > -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to emscripten-discuss+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Remain an unused import function in wasm
I declare an external function in C source file, but never used by the code, then compile it to wasm by EMCC. there is no doubt the external function will be eliminated by the compiler. Is there any approach to keep this external function and then be compiled to an import function in wasm? The motivation to remain the external function is that I want to insert some wasm code in the wasm file after it's generated. Thanks all! -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Can I generate the external function declaration in the specific module?
Thanks. I have done the process that by manual, worked but not concise, this is why I posted this thread. On the other hand, I don't find any spec about if we can or can't generate the user-defined module. I hope we can have a better approach to do it. 在 2019年2月15日星期五 UTC+8上午3:09:10，Alon Zakai写道： > > There isn't a good way to do that right now, but you can use wabt or > binaryen to emit wasm text, edit that, and convert back to wasm. > > (It's almost possible to do this in binaryen.js - we have the tools to get > the function's module, but not set it, yet.) > > On Tue, Feb 12, 2019 at 11:09 PM Duan Bing > > wrote: > >> I write a C source file, declare an external function by >> extern "C" int say_hello(const char*, int); >> >> So after compiling by `emcc`, I got >> >> (import "env" "_say_hello" (func (;30;) (type 1))) >> >> in the math.wast. Everything is ok for now. but I want to change the >> module of _say_hello to another name, like "mymodule". >> >> How can I approach that? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "emscripten-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to emscripten-discuss+unsubscr...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Can I generate the external function declaration in the specific module?
I write a C source file, declare an external function by extern "C" int say_hello(const char*, int); So after compiling by `emcc`, I got (import "env" "_say_hello" (func (;30;) (type 1))) in the math.wast. Everything is ok for now. but I want to change the module of _say_hello to another name, like "mymodule". How can I approach that? -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
