[
https://issues.apache.org/jira/browse/ARROW-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872219#comment-16872219
]
Liya Fan commented on ARROW-5658:
---------------------------------
[~liaotian1005] we have found the reason for the failure.
It is caused by inconsistent schema for VectorSchemaRoot.
In particular, the schema looks like this at the beginning:
!image-2019-06-25-17-58-09-038.png!
As new data are inserted to the vector, vector structure changes to something
like this:
!image-2019-06-25-17-59-07-352.png!
The change is due to a mechanism named writer promotion. For details, please
see class PromotableWriter.
In the client side, the new vector structure is sent along with the obsolete
schema. In the server side, the code tries to decode the vector structure with
the out-of-date schema, so it fails, without printing any output.
Our solution is to provide a method in class VectorSchemaRoot, which brings the
schema in sync with the vector structure. So to fix the problem in your code,
simply call this method after the data is inserted. Please see the modified
code here: [^ClientStart.java]
> [JAVA] apache arrow-flight cannot send listvector
> --------------------------------------------------
>
> Key: ARROW-5658
> URL: https://issues.apache.org/jira/browse/ARROW-5658
> Project: Apache Arrow
> Issue Type: Bug
> Components: FlightRPC, Java
> Affects Versions: 0.13.0
> Environment: java8 arrow-java 0.13.0
> Reporter: luckily
> Assignee: Liya Fan
> Priority: Major
> Labels: pull-request-available
> Attachments: ClientStart.java, ClientStart.java, ServerStart.java,
> image-2019-06-25-17-58-09-038.png, image-2019-06-25-17-59-07-352.png, pom.xml
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I can't transfer using apache arrow-flihgt. Contains listvector data. The
> problem description is as follows:
> {quote} # I parse an xml file and convert it to an arrow format and finally
> convert it to a parquet data format. The address of the .xml file data is url
> [http://www.w3school.com.cn/example/xmle/cd_catalog.xml|http://www.w3school.com.cn/example/xmle/cd_catalog.xml)]
> # I created a schema that uses listvector.
> code show as below:
> List<FiledVector> list =
> childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
> VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
> # Parse the xml file to get the list data in "cd". Use api use listvector.
> `ListVector listVector = (ListVector) valueVectors;
> List<Column> columns = column.getColumns();
> Column column1 = columns.get(0);
> String name = column1.getId().toString();
> UnionListWriter writer = listVector.getWriter();
> Writer.allocate();
> For (int j = 0; j < column1.getColumns().size();j++) {
> writer.setPosition(j);
> writer.startList();
> Writer.list().startList();
> Column column2 = column1.getColumns().get(j);
> List<Map<String, String>> lst = (List<Map<String,
> String>>) ((Map) val).get(name);
> For (int k = 0; k < lst.size(); k++) {
> Map<String, String> stringStringMap = lst.get(k);
> String value =
> stringStringMap.get(column2.getId().toString());
> Switch (column2.getType()) {
> Case FLOAT:
>
> Writer.list().float4().writeFloat4(stringConvertFloat(value));
> Break;
> Case BOOLEAN:
>
> Writer.list().bit().writeBit(stringConvertBoolean(value));
> Break;
> Case DECIMAL:
>
> Writer.list().decimal().writeDecimal(stringConvertDecimal(value,column2.getScale()));
> Break;
> Case TIMESTAMP:
>
> Writer.list().dateMilli().writeDateMilli(stringConvertTimestamp(value,column2.format.toString()));
> Break;
> Case INTEGER:
> Case BIGINT:
>
> Writer.list().bigInt().writeBigInt(stringConvertLong(value));
> Break;
> Case VARCHAR:
> VarCharHolder varBinaryHolder = new
> VarCharHolder();
> varBinaryHolder.start = 0;
> Byte[] bytes =value.getBytes();
> ArrowBuf buffer =
> listVector.getAllocator().buffer(bytes.length);
> varBinaryHolder.buffer = buffer;
> buffer.writeBytes(bytes);
> varBinaryHolder.end=bytes.length;
>
> Writer.list().varChar().write(varBinaryHolder);
> Break;
> Default:
> Throw new IllegalArgumentException(" error no
> type !!");
> }
> }
> Writer.list().endList();
> writer.endList();
> }`
> 4.
> After the write is complete, I will send to the arrow-flight server. server
> code :
> {quote}
> {quote}@Override
> public Callable<Flight.PutResult> acceptPut(FlightStream flightStream) {
> return () -> {
> try (VectorSchemaRoot root = flightStream.getRoot()) {
> while (flightStream.next()) {
> VectorSchemaRoot other = null;
> try {
> logger.info(" Receive message ...... size: " + root.getRowCount());
> other = copyRoot(root);
> ArrowMessage arrowMessage = new ArrowMessage(other, other.getSchema());
> spmc.offer(arrowMessage);
> } catch (Exception e) {
> logger.error(e.getMessage(), e);
> }
> }
> }
> return Flight.PutResult.parseFrom("ok".getBytes());
> };
> }{quote}
> {quote} But the server did not receive any information.!! it is error .{quote}
> {quote}client code :{quote}
> {quote}root = message.getRoot();
> //client.close();
> FlightClient.ClientStreamListener listener =
> client.startPut(FlightDescriptor.path(message.getFilename()), root);
> listener.putNext();
> listener.completed();
> client.close();
> listener.putNext();
> listener.completed();
> Flight.PutResult result =
> listener.getResult();
> String s = new String(result.toByteArray());
> System.out.println(s);{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
