Re: Some question about document

2018-07-16 Thread Yuta Morisawa 

Hi yang

Thank you for your comment.
I read document and got an idea.

Flink uses custom serializers on known types, and the fallback is kryo.
The other arbitary objects is serialized by kryo.



On 2018/07/12 12:14, vino yang wrote:

Hi Yuta,

It seems Chesnay is right. The "fallback" in flink's documentation is in 
terms of the types flink supported. But for all the other arbitrary 
types kryo is the first choice.


2018-07-12 9:55 GMT+08:00 Yuta Morisawa >:


Thank you for your answer.

> For POJOs Flink has a custom serializer. For arbitrary objects we use
> kryo, and can use Avro as a fallback.

https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types



It may be the reverse.
Kryo is for fallback, right?


On 2018/07/11 19:00, Chesnay Schepler wrote:

1) TypeInformation are used to create serializers, comparators
and to verify correctness of certain operations (like
projections on tuple datasets).

2) see

https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html



3) Flink comes with a number of serializers for varying types as
outlined here

>.
For POJOs Flink has a custom serializer. For arbitrary objects
we use kryo, and can use Avro as a fallback.

On 11.07.2018 09:24, Yuta Morisawa wrote:

Hi all

Now, I'm reading Flink document and I have some points to
feel difficult to get an idea.
I'd appreciate if you tell it me.

1,TypeInformation
  I understand TypeInformation is used for selecting
relevant serializer and comparator.
  But, the ducument doesn't specify if it has another way to
be used.

  So, what I want to know is that what kinds of process gets
benefit from TypeInformation other than serializer and
comparator.

2, Managed Memory
  The word "Managed memory" is appeared several time in the
document but I can't find any detail description.
  This is the only document I found

(https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi

)

  If anyone has document that explains managed memory,
please let me know.

3, Serializer
  What do the words in the document  "serializers we ship
with Flink" mean? I know Flink uses avro for POJOs, is it
the same thing?

https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html





Regards,
Yuta

Re: Some question about document

2018-07-11 Thread vino yang 
Hi Yuta,

It seems Chesnay is right. The "fallback" in flink's documentation is in
terms of the types flink supported. But for all the other arbitrary types
kryo is the first choice.

2018-07-12 9:55 GMT+08:00 Yuta Morisawa :

> Thank you for your answer.
>
> > For POJOs Flink has a custom serializer. For arbitrary objects we use
> > kryo, and can use Avro as a fallback.
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/
> dev/types_serialization.html#serialization-of-pojo-types
>
> It may be the reverse.
> Kryo is for fallback, right?
>
>
> On 2018/07/11 19:00, Chesnay Schepler wrote:
>
>> 1) TypeInformation are used to create serializers, comparators and to
>> verify correctness of certain operations (like projections on tuple
>> datasets).
>>
>> 2) see https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-
>> and-Bytes.html
>>
>> 3) Flink comes with a number of serializers for varying types as outlined
>> here > 5/dev/types_serialization.html#flinks-typeinformation-class>.
>> For POJOs Flink has a custom serializer. For arbitrary objects we use
>> kryo, and can use Avro as a fallback.
>>
>> On 11.07.2018 09:24, Yuta Morisawa wrote:
>>
>>> Hi all
>>>
>>> Now, I'm reading Flink document and I have some points to feel difficult
>>> to get an idea.
>>> I'd appreciate if you tell it me.
>>>
>>> 1,TypeInformation
>>>  I understand TypeInformation is used for selecting relevant serializer
>>> and comparator.
>>>  But, the ducument doesn't specify if it has another way to be used.
>>>
>>>  So, what I want to know is that what kinds of process gets benefit from
>>> TypeInformation other than serializer and comparator.
>>>
>>> 2, Managed Memory
>>>  The word "Managed memory" is appeared several time in the document but
>>> I can't find any detail description.
>>>  This is the only document I found (https://www.slideshare.net/sb
>>> altagi/overview-of-apacheflinkbyslimbaltagi)
>>>
>>>  If anyone has document that explains managed memory, please let me know.
>>>
>>> 3, Serializer
>>>  What do the words in the document  "serializers we ship with Flink"
>>> mean? I know Flink uses avro for POJOs, is it the same thing?
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/
>>> dev/types_serialization.html
>>>
>>>
>>> Regards,
>>> Yuta
>>>
>>>
>>
>

Re: Some question about document

2018-07-11 Thread Yuta Morisawa 

Thank you for your answer.

> For POJOs Flink has a custom serializer. For arbitrary objects we use
> kryo, and can use Avro as a fallback.
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types

It may be the reverse.
Kryo is for fallback, right?


On 2018/07/11 19:00, Chesnay Schepler wrote:
1) TypeInformation are used to create serializers, comparators and to 
verify correctness of certain operations (like projections on tuple 
datasets).


2) see 
https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html


3) Flink comes with a number of serializers for varying types as 
outlined here 
.
For POJOs Flink has a custom serializer. For arbitrary objects we use 
kryo, and can use Avro as a fallback.


On 11.07.2018 09:24, Yuta Morisawa wrote:

Hi all

Now, I'm reading Flink document and I have some points to feel 
difficult to get an idea.

I'd appreciate if you tell it me.

1,TypeInformation
 I understand TypeInformation is used for selecting relevant 
serializer and comparator.

 But, the ducument doesn't specify if it has another way to be used.

 So, what I want to know is that what kinds of process gets benefit 
from TypeInformation other than serializer and comparator.


2, Managed Memory
 The word "Managed memory" is appeared several time in the document 
but I can't find any detail description.
 This is the only document I found 
(https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)


 If anyone has document that explains managed memory, please let me know.

3, Serializer
 What do the words in the document  "serializers we ship with Flink" 
mean? I know Flink uses avro for POJOs, is it the same thing?
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html 




Regards,
Yuta

Re: Some question about document

2018-07-11 Thread Chesnay Schepler 
1) TypeInformation are used to create serializers, comparators and to 
verify correctness of certain operations (like projections on tuple 
datasets).


2) see 
https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html


3) Flink comes with a number of serializers for varying types as 
outlined here 
.
For POJOs Flink has a custom serializer. For arbitrary objects we use 
kryo, and can use Avro as a fallback.


On 11.07.2018 09:24, Yuta Morisawa wrote:

Hi all

Now, I'm reading Flink document and I have some points to feel 
difficult to get an idea.

I'd appreciate if you tell it me.

1,TypeInformation
 I understand TypeInformation is used for selecting relevant 
serializer and comparator.

 But, the ducument doesn't specify if it has another way to be used.

 So, what I want to know is that what kinds of process gets benefit 
from TypeInformation other than serializer and comparator.


2, Managed Memory
 The word "Managed memory" is appeared several time in the document 
but I can't find any detail description.
 This is the only document I found 
(https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)


 If anyone has document that explains managed memory, please let me know.

3, Serializer
 What do the words in the document  "serializers we ship with Flink" 
mean? I know Flink uses avro for POJOs, is it the same thing?
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html 




Regards,
Yuta

Some question about document

2018-07-11 Thread Yuta Morisawa 

Hi all

Now, I'm reading Flink document and I have some points to feel difficult 
to get an idea.

I'd appreciate if you tell it me.

1,TypeInformation
 I understand TypeInformation is used for selecting relevant serializer 
and comparator.

 But, the ducument doesn't specify if it has another way to be used.

 So, what I want to know is that what kinds of process gets benefit 
from TypeInformation other than serializer and comparator.


2, Managed Memory
 The word "Managed memory" is appeared several time in the document 
but I can't find any detail description.
 This is the only document I found 
(https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)


 If anyone has document that explains managed memory, please let me know.

3, Serializer
 What do the words in the document  "serializers we ship with Flink" 
mean? I know Flink uses avro for POJOs, is it the same thing?

https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html


Regards,
Yuta

--


 Business Vision ："Challenge for the future"
-

  〒356-8502
　埼玉県ふじみ野市大原2丁目1番15号
　株式会社　ＫＤＤＩ総合研究所（KDDI Research, Inc.）
　コネクティッドカー1G
　森澤　雄太
　mail yu-moris...@kddi-research.jp
　tel  070-3871-8883

　この電子メールおよび添付書類は、名宛人のための
　特別な秘密情報を含んでおります。
　そのため、名宛人以外の方による利用は認められて
　おりません。
　名宛人以外の方による通信内容公表、複写、転用等
　は厳禁であり、違法となることがあります。
　万が一、何らかの誤りによりこの電子メールを名宛
　人以外の方が受信された場合は、お手数でも、直ち
　に発信人にお知らせ頂くと同時に、当メールを削除
　下さいますようお願い申し上げます。