[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874801#comment-16874801
 ] 

Liya Fan commented on ARROW-5658:
-

[~liaotian1005] we have changed the code to automatically update the schema, so 
you do not need to call syncSchema in your code. 

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ClientStart.java, 
> ServerStart.java, ServerStart.java, ServerStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, 
> image-2019-06-27-09-38-52-735.png, image-2019-06-27-09-39-18-597.png, pom.xml
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874659#comment-16874659
 ] 

luckily commented on ARROW-5658:


I don't think the data in position in listvector should have anything to do 
with rowcount.

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ClientStart.java, 
> ServerStart.java, ServerStart.java, ServerStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, 
> image-2019-06-27-09-38-52-735.png, image-2019-06-27-09-39-18-597.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874656#comment-16874656
 ] 

luckily commented on ARROW-5658:


I have two vector, one is varchar, the other is list.. Assuming that varchar 
has two pieces of data and list has three postion, how much rowcount should be 
set?

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ClientStart.java, 
> ServerStart.java, ServerStart.java, ServerStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, 
> image-2019-06-27-09-38-52-735.png, image-2019-06-27-09-39-18-597.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874600#comment-16874600
 ] 

Liya Fan commented on ARROW-5658:
-

SetRowCount determines the range of indices that can be used without causing an 
IndexOutOfBoundException

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ClientStart.java, 
> ServerStart.java, ServerStart.java, ServerStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, 
> image-2019-06-27-09-38-52-735.png, image-2019-06-27-09-39-18-597.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874580#comment-16874580
 ] 

luckily commented on ARROW-5658:


SetRowCount determines that list can get index data?

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ClientStart.java, 
> ServerStart.java, ServerStart.java, ServerStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, 
> image-2019-06-27-09-38-52-735.png, image-2019-06-27-09-39-18-597.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873902#comment-16873902
 ] 

Liya Fan commented on ARROW-5658:
-

Hi [~liaotian1005], thanks for the additional information.

I did not reproduce the problem as described by you. When I set the row count 
to 1, 2 or 3, the server could receive the record batch correctly:

When row count was set to 1, the server receives the batch with only 1 row, and 
the second row is truncated.

When row count was set to 2, the server receives the batch with 2 rows, as 2 
rows were sent from the client.

When row count was set to 3, the server receives the batch with 3 rows, but the 
3rd row is null, as there were only 2 rows from the client side. 

To help you reproduce the above results, I have attached the code for my server 
side. Please try it. [^ServerStart.java]

 

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ClientStart.java, 
> ServerStart.java, ServerStart.java, ServerStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, 
> image-2019-06-27-09-38-52-735.png, image-2019-06-27-09-39-18-597.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873747#comment-16873747
 ] 

luckily commented on ARROW-5658:


hi [~fan_li_ya]  ,Why when using root.setRowCount (3), the server can accept 
the complete data. When using root.setRowCount (2) or root.setRowCount (1). The 
server did not receive the complete data, Why setRowCount this api determines 
whether the listat can accept the full data?

!image-2019-06-27-09-38-52-735.png!

server 

!image-2019-06-27-09-39-18-597.png!

 

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, 
> image-2019-06-27-09-38-52-735.png, image-2019-06-27-09-39-18-597.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
>  

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873269#comment-16873269
 ] 

Liya Fan commented on ARROW-5658:
-

Hi [~liaotian1005], I don't quite understand the new problem.

Would you please provide the code to reproduce the problem?

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873179#comment-16873179
 ] 

luckily commented on ARROW-5658:


Data url [http://www.w3school.com.cn/example/xmle/cd_catalog.xml]   cd.xml. 

I write the data in url to a VectorSchemaRoot, to look at this file from a 
global point of view. Is a piece of data, one of which includes a list,list. 
The number of bars is 24 (bars). The index is 6 (6 columns). So I send the data 
when the parsing is complete. VectorSchemaRoot.setvalueCount (1) 
listvector.setValueCount (6), but I didn't accept listVector data on the 
server. At the same time, I found out. When VectorSchemaRoot.setvalueCount (6), 
the server can accept the complete listVector data. But from a global point of 
view, there is only one, so I call VectorSchemaRoot.setvalueCount (1), but the 
server does not accept the data. Despite the use of listvector.setValueCount 
(6), why does the server receive data when VectorSchemaRoot.setvalueCount (6) 
is used?

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873138#comment-16873138
 ] 

Liya Fan commented on ARROW-5658:
-

Hi [~liaotian1005], please check if our patch fixes your new problem. If not, 
it may be appropriate to open a separate Jira to track it.

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.13.0
>
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, pom.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873076#comment-16873076
 ] 

luckily commented on ARROW-5658:


Hi LiyaFan 

i pull you code 

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Major
>  Labels: pull-request-available
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, pom.xml
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873060#comment-16873060
 ] 

luckily commented on ARROW-5658:


Hi LiyaFan 

I need to do an asynchronous process to prevent the IO thread from blocking. In 
many cases, the row data is not immediately processed, placed in an 
asynchronous queue, but cannot be placed directly. It is necessary to create a 
same container, if there is such an api, you can copy Root once, Create a same 
data

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Major
>  Labels: pull-request-available
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, pom.xml
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
>    

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873055#comment-16873055
 ] 

luckily commented on ARROW-5658:


[~fan_li_ya]

I've got a new problem, that's the loss of data.

I created a ListVector.. There are six index. in the data. Data such as the xml 
file mentioned above. There are six columns.

!image-2019-06-26-16-07-23-366.png!

After I send the data, the server can only receive the data with the index of 
0, and the other index locations are all missing.

!image-2019-06-26-16-09-30-275.png!

 

 

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Major
>  Labels: pull-request-available
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, 
> image-2019-06-26-16-07-23-366.png, image-2019-06-26-16-09-30-275.png, pom.xml
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873046#comment-16873046
 ] 

luckily commented on ARROW-5658:


I've got a new problem, that's the loss of data.

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Major
>  Labels: pull-request-available
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, pom.xml
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try {
>  

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872219#comment-16872219
 ] 

Liya Fan commented on ARROW-5658:
-

[~liaotian1005] we have found the reason for the failure.

It is caused by inconsistent schema for VectorSchemaRoot.

In particular, the schema looks like this at the beginning:

!image-2019-06-25-17-58-09-038.png!

As new data are inserted to the vector, vector structure changes to something 
like this:

!image-2019-06-25-17-59-07-352.png!

The change is due to a mechanism named writer promotion. For details, please 
see class PromotableWriter.

In the client side, the new vector structure is sent along with the obsolete 
schema. In the server side, the code tries to decode the vector structure with 
the out-of-date schema, so it fails, without printing any output. 

 

Our solution is to provide a method in class VectorSchemaRoot, which brings the 
schema in sync with the vector structure. So to fix the problem in your code, 
simply call this method after the data is inserted. Please see the modified 
code here: [^ClientStart.java]

 

 

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Major
>  Labels: pull-request-available
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java, 
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, pom.xml
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871023#comment-16871023
 ] 

Liya Fan commented on ARROW-5658:
-

[~liaotian1005] thanks for the additional information.

We have found the root cause, and is in the process of fixing it. The problem 
seems to be with the serialization/deserizlization process. 

When there is any progress, we will let you know about it.

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Assignee: Liya Fan
>Priority: Major
> Attachments: ClientStart.java, ServerStart.java, pom.xml
>
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870734#comment-16870734
 ] 

luckily commented on ARROW-5658:


[~lidavidm],arrow stream api cannot send listvector too

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Major
> Attachments: ClientStart.java, ServerStart.java, pom.xml
>
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try {
>  logger.info(" Receive message .. size: " + root.getRowCount());
>  other = copyRoot(root);
>  ArrowMessage arrowMessage = new ArrowMessage(other, other.getSchema());
>  spmc.offer(arrowMessage);
>  } catch (Exception e) {
>  

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870732#comment-16870732
 ] 

luckily commented on ARROW-5658:


[~lidavidm] ,The flight server does not receive any messages, nor does it have 
an error message prompt

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Major
> Attachments: ClientStart.java, ServerStart.java, pom.xml
>
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try {
>  logger.info(" Receive message .. size: " + root.getRowCount());
>  other = copyRoot(root);
>  ArrowMessage arrowMessage = new ArrowMessage(other, other.getSchema());
>  

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[lidavidm (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869421#comment-16869421
 ] 

lidavidm commented on ARROW-5658:
-

[~liaotian1005], generally UNKNOWN means that there was an uncaught server-side 
exception. Do you see any traceback in the server output? If not, then we need 
to log these things inside Flight.

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Major
> Attachments: ClientStart.java, ServerStart.java, pom.xml
>
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try {
>  logger.info(" Receive message .. size: " + root.getRowCount());
>  other = 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869346#comment-16869346
 ] 

luckily commented on ARROW-5658:


Exception in thread "main" java.lang.RuntimeException: 
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNKNOWN
 at com.google.common.base.Throwables.propagate(Throwables.java:241)
 at 
org.apache.arrow.flight.FlightClient$PutObserver.getResult(FlightClient.java:244)
 at ClientStart.main(ClientStart.java:45)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: UNKNOWN
 at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:528)
 at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509)
 at 
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:85)
 at 
org.apache.arrow.flight.FlightClient$PutObserver.getResult(FlightClient.java:242)
 ... 1 more
Caused by: io.grpc.StatusRuntimeException: UNKNOWN
 at io.grpc.Status.asRuntimeException(Status.java:526)
 at 
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
 at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
 at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
 at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
 at 
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
 at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
 at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
 at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
 at 
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
 at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
 at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
 at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
 at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
 at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Major
> Attachments: ClientStart.java, ServerStart.java, pom.xml
>
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[luckily (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869344#comment-16869344
 ] 

luckily commented on ARROW-5658:


Hi  I've written an example about the error, in the attachment, I put forward 
my opinion, whether the list write the varchar data is too much trouble, how to 
use the list, whether to give a detailed document, whether to support the 
ListVector , this general structure, if this structure does 
not support the image xml. And the json file format is converted to a row 
format. Thank you. I've written an example about the error, in the attachment, 
I put forward my opinion, whether the list write the varchar data is too much 
trouble, how to use the list, whether to give a detailed document, whether to 
support the ListVector , this general structure, if this 
structure does not support the image xml. And the json file format is converted 
to a row format. Thank you.

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Major
> Attachments: ClientStart.java, ServerStart.java, pom.xml
>
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> 

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869172#comment-16869172
 ] 

Liya Fan commented on ARROW-5658:
-

Hi [~liaotian1005], so far I have failed to reproduce the problem.

Would you please provide more code to reproduce the problem? Maybe you can 
upload the related code to github?

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Major
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try {
>  logger.info(" Receive message .. size: " + root.getRowCount());
>  other = copyRoot(root);
>  ArrowMessage arrowMessage = new ArrowMessage(other, other.getSchema());
>  

[jira] [Commented] (ARROW-5658) [JAVA] apache arrow-flight cannot send listvector

[Micah Kornfield (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868319#comment-16868319
 ] 

Micah Kornfield commented on ARROW-5658:


What is message defined as in the code above?  It seems like parquet might be 
complicating the matters, did you try creating a simple in memory Vector to 
send? 

> [JAVA] apache arrow-flight cannot send listvector 
> --
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: FlightRPC, Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Major
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try {
>  logger.info(" Receive message .. size: " + root.getRowCount());
>  other = copyRoot(root);
>  ArrowMessage arrowMessage = new ArrowMessage(other, other.getSchema());
>  

[jira] [Commented] (ARROW-5658) [JAVA]apache arrow-flight cannot send listvector

[Liya Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868294#comment-16868294
 ] 

Liya Fan commented on ARROW-5658:
-

Hi [~liaotian1005] thanks a lot for reporting the problem and the detailed 
information.

I am not familiar with arrow-flight, but will take a look at this issue. 

> [JAVA]apache arrow-flight cannot send listvector 
> -
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: Java
>Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
>Reporter: luckily
>Priority: Blocker
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The 
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally 
> convert it to a parquet data format. The address of the .xml file data is url 
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
>  # I created a schema that uses listvector.
> code show as below:
> List list = 
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
>  # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List> lst = (List String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map stringStringMap = lst.get(k);
> String value = 
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
> 
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
> 
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
> 
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
> 
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
> 
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new 
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer = 
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
> 
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no 
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
>  4. 
> After the write is complete, I will send to the arrow-flight server. server 
> code :
> {quote}
> {quote}@Override
> public Callable acceptPut(FlightStream flightStream) {
>  return () -> {
>  try (VectorSchemaRoot root = flightStream.getRoot()) {
>  while (flightStream.next()) {
>  VectorSchemaRoot other = null;
>  try {
>  logger.info(" Receive message .. size: " + root.getRowCount());
>  other = copyRoot(root);
>  ArrowMessage arrowMessage = new ArrowMessage(other, other.getSchema());
>  spmc.offer(arrowMessage);