[GitHub] [incubator-hudi] bhasudha commented on a change in pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bhasudha commented on a change in pull request #929: [HUDI-271] Create 
QuickstartUtils class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929#discussion_r329338863
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java
 ##
 @@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.Deflater;
+import java.util.zip.DeflaterOutputStream;
+import java.util.zip.InflaterInputStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.MercifulJsonConverter;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+/**
+ * Class to be used in quickstart guide for generating inserts and updates 
against a corpus.
+ * 
+ * Test data uses a toy Uber trips, data model.
+ *
+ */
+public class QuickstartUtils {
+
+  public static class DataGenerator {
+private static final String DEFAULT_FIRST_PARTITION_PATH = "2019/09/15";
+private static final String DEFAULT_SECOND_PARTITION_PATH = "2018/09/16";
+private static final String DEFAULT_THIRD_PARTITION_PATH = "2018/09/17";
+
+private static final String[] DEFAULT_PARTITION_PATHS = {
+DEFAULT_FIRST_PARTITION_PATH,
+DEFAULT_SECOND_PARTITION_PATH,
+DEFAULT_THIRD_PARTITION_PATH
+};
+static String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": 
\"triprec\"," + "\"fields\": [ "
++ "{\"name\": \"ts\",\"type\": \"double\"},"
++ "{\"name\": \"uuid\", \"type\": \"string\"},"
++ "{\"name\": \"rider\", \"type\": \"string\"},"
++ "{\"name\": \"driver\", \"type\": \"string\"},"
++ "{\"name\": \"begin_lat\", \"type\": \"double\"},"
++ "{\"name\": \"begin_lon\", \"type\": \"double\"},"
++ "{\"name\": \"end_lat\", \"type\": \"double\"},"
++ "{\"name\": \"end_lon\", \"type\": \"double\"},"
++ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+static Schema avroSchema = new Schema.Parser().parse(TRIP_EXAMPLE_SCHEMA);
+
+private static Random rand = new Random(46474747);
+
+private final Map existingKeys;
+private final String[] partitionPaths;
+private int numExistingKeys;
+
+public DataGenerator() {
+  this(DEFAULT_PARTITION_PATHS, new HashMap<>());
+}
+
+private DataGenerator(String[] partitionPaths, Map 
keyPartitionMap) {
+  this.partitionPaths = Arrays.copyOf(partitionPaths, 
partitionPaths.length);
+  this.existingKeys = keyPartitionMap;
+}
+
+private static String generateRandomCommitMsg() {
+  int leftLimit = 48; // ascii for 0
+  int rightLimit = 57; // ascii for 9
+  int stringLength = 3;
+  StringBuilder buffer = new StringBuilder(stringLength);
+  for (int i = 0; i < stringLength; i++) {
+int randomLimitedInt = leftLimit + (int)
+(rand.nextFloat() * (rightLimit - leftLimit + 1));
+buffer.append((char) randomLimitedInt);
+  }
+  return buffer.toString();
+}
+
+public int getNumExistingKeys() {
+  return numExistingKeys;
+}
+
+public static GenericRecord generateGenericRecord(String rowKey, String 
riderName, String driverName,
+  double timestamp) {
+  GenericRecord rec = new GenericData.Record(avroSchema);
+  rec.put

[GitHub] [incubator-hudi] bhasudha commented on a change in pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bhasudha commented on a change in pull request #929: [HUDI-271] Create 
QuickstartUtils class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929#discussion_r329338796
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java
 ##
 @@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.Deflater;
+import java.util.zip.DeflaterOutputStream;
+import java.util.zip.InflaterInputStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.MercifulJsonConverter;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+/**
+ * Class to be used in quickstart guide for generating inserts and updates 
against a corpus.
+ * 
+ * Test data uses a toy Uber trips, data model.
+ *
+ */
+public class QuickstartUtils {
+
+  public static class DataGenerator {
+private static final String DEFAULT_FIRST_PARTITION_PATH = "2019/09/15";
+private static final String DEFAULT_SECOND_PARTITION_PATH = "2018/09/16";
+private static final String DEFAULT_THIRD_PARTITION_PATH = "2018/09/17";
+
+private static final String[] DEFAULT_PARTITION_PATHS = {
+DEFAULT_FIRST_PARTITION_PATH,
+DEFAULT_SECOND_PARTITION_PATH,
+DEFAULT_THIRD_PARTITION_PATH
+};
+static String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": 
\"triprec\"," + "\"fields\": [ "
++ "{\"name\": \"ts\",\"type\": \"double\"},"
++ "{\"name\": \"uuid\", \"type\": \"string\"},"
++ "{\"name\": \"rider\", \"type\": \"string\"},"
++ "{\"name\": \"driver\", \"type\": \"string\"},"
++ "{\"name\": \"begin_lat\", \"type\": \"double\"},"
++ "{\"name\": \"begin_lon\", \"type\": \"double\"},"
++ "{\"name\": \"end_lat\", \"type\": \"double\"},"
++ "{\"name\": \"end_lon\", \"type\": \"double\"},"
++ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+static Schema avroSchema = new Schema.Parser().parse(TRIP_EXAMPLE_SCHEMA);
+
+private static Random rand = new Random(46474747);
+
+private final Map existingKeys;
+private final String[] partitionPaths;
+private int numExistingKeys;
+
+public DataGenerator() {
+  this(DEFAULT_PARTITION_PATHS, new HashMap<>());
+}
+
+private DataGenerator(String[] partitionPaths, Map 
keyPartitionMap) {
+  this.partitionPaths = Arrays.copyOf(partitionPaths, 
partitionPaths.length);
+  this.existingKeys = keyPartitionMap;
+}
+
+private static String generateRandomCommitMsg() {
 
 Review comment:
   done!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] bhasudha commented on a change in pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bhasudha commented on a change in pull request #929: [HUDI-271] Create 
QuickstartUtils class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929#discussion_r329338800
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java
 ##
 @@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.Deflater;
+import java.util.zip.DeflaterOutputStream;
+import java.util.zip.InflaterInputStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.MercifulJsonConverter;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+/**
+ * Class to be used in quickstart guide for generating inserts and updates 
against a corpus.
+ * 
+ * Test data uses a toy Uber trips, data model.
+ *
+ */
+public class QuickstartUtils {
+
+  public static class DataGenerator {
+private static final String DEFAULT_FIRST_PARTITION_PATH = "2019/09/15";
+private static final String DEFAULT_SECOND_PARTITION_PATH = "2018/09/16";
+private static final String DEFAULT_THIRD_PARTITION_PATH = "2018/09/17";
+
+private static final String[] DEFAULT_PARTITION_PATHS = {
+DEFAULT_FIRST_PARTITION_PATH,
+DEFAULT_SECOND_PARTITION_PATH,
+DEFAULT_THIRD_PARTITION_PATH
+};
+static String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": 
\"triprec\"," + "\"fields\": [ "
++ "{\"name\": \"ts\",\"type\": \"double\"},"
++ "{\"name\": \"uuid\", \"type\": \"string\"},"
++ "{\"name\": \"rider\", \"type\": \"string\"},"
++ "{\"name\": \"driver\", \"type\": \"string\"},"
++ "{\"name\": \"begin_lat\", \"type\": \"double\"},"
++ "{\"name\": \"begin_lon\", \"type\": \"double\"},"
++ "{\"name\": \"end_lat\", \"type\": \"double\"},"
++ "{\"name\": \"end_lon\", \"type\": \"double\"},"
++ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+static Schema avroSchema = new Schema.Parser().parse(TRIP_EXAMPLE_SCHEMA);
+
+private static Random rand = new Random(46474747);
+
+private final Map existingKeys;
+private final String[] partitionPaths;
+private int numExistingKeys;
+
+public DataGenerator() {
+  this(DEFAULT_PARTITION_PATHS, new HashMap<>());
+}
+
+private DataGenerator(String[] partitionPaths, Map 
keyPartitionMap) {
+  this.partitionPaths = Arrays.copyOf(partitionPaths, 
partitionPaths.length);
+  this.existingKeys = keyPartitionMap;
+}
+
+private static String generateRandomCommitMsg() {
+  int leftLimit = 48; // ascii for 0
+  int rightLimit = 57; // ascii for 9
+  int stringLength = 3;
+  StringBuilder buffer = new StringBuilder(stringLength);
+  for (int i = 0; i < stringLength; i++) {
+int randomLimitedInt = leftLimit + (int)
+(rand.nextFloat() * (rightLimit - leftLimit + 1));
+buffer.append((char) randomLimitedInt);
+  }
+  return buffer.toString();
+}
+
+public int getNumExistingKeys() {
+  return numExistingKeys;
+}
+
+public static GenericRecord generateGenericRecord(String rowKey, String 
riderName, String driverName,
+  double timestamp) {
+  GenericRecord rec = new GenericData.Record(avroSchema);
+  rec.put

Build failed in Jenkins: hudi-snapshot-deployment-0.5 #52

2019-09-28 Thread Apache Jenkins Server 
See 


--
[...truncated 2.17 KB...]
/home/jenkins/tools/maven/apache-maven-3.5.4/bin:
m2.conf
mvn
mvn.cmd
mvnDebug
mvnDebug.cmd
mvnyjp

/home/jenkins/tools/maven/apache-maven-3.5.4/boot:
plexus-classworlds-2.5.2.jar

/home/jenkins/tools/maven/apache-maven-3.5.4/conf:
logging
settings.xml
toolchains.xml

/home/jenkins/tools/maven/apache-maven-3.5.4/conf/logging:
simplelogger.properties

/home/jenkins/tools/maven/apache-maven-3.5.4/lib:
aopalliance-1.0.jar
cdi-api-1.0.jar
cdi-api.license
commons-cli-1.4.jar
commons-cli.license
commons-io-2.5.jar
commons-io.license
commons-lang3-3.5.jar
commons-lang3.license
ext
guava-20.0.jar
guice-4.2.0-no_aop.jar
jansi-1.17.1.jar
jansi-native
javax.inject-1.jar
jcl-over-slf4j-1.7.25.jar
jcl-over-slf4j.license
jsr250-api-1.0.jar
jsr250-api.license
maven-artifact-3.5.4.jar
maven-artifact.license
maven-builder-support-3.5.4.jar
maven-builder-support.license
maven-compat-3.5.4.jar
maven-compat.license
maven-core-3.5.4.jar
maven-core.license
maven-embedder-3.5.4.jar
maven-embedder.license
maven-model-3.5.4.jar
maven-model-builder-3.5.4.jar
maven-model-builder.license
maven-model.license
maven-plugin-api-3.5.4.jar
maven-plugin-api.license
maven-repository-metadata-3.5.4.jar
maven-repository-metadata.license
maven-resolver-api-1.1.1.jar
maven-resolver-api.license
maven-resolver-connector-basic-1.1.1.jar
maven-resolver-connector-basic.license
maven-resolver-impl-1.1.1.jar
maven-resolver-impl.license
maven-resolver-provider-3.5.4.jar
maven-resolver-provider.license
maven-resolver-spi-1.1.1.jar
maven-resolver-spi.license
maven-resolver-transport-wagon-1.1.1.jar
maven-resolver-transport-wagon.license
maven-resolver-util-1.1.1.jar
maven-resolver-util.license
maven-settings-3.5.4.jar
maven-settings-builder-3.5.4.jar
maven-settings-builder.license
maven-settings.license
maven-shared-utils-3.2.1.jar
maven-shared-utils.license
maven-slf4j-provider-3.5.4.jar
maven-slf4j-provider.license
org.eclipse.sisu.inject-0.3.3.jar
org.eclipse.sisu.inject.license
org.eclipse.sisu.plexus-0.3.3.jar
org.eclipse.sisu.plexus.license
plexus-cipher-1.7.jar
plexus-cipher.license
plexus-component-annotations-1.7.1.jar
plexus-component-annotations.license
plexus-interpolation-1.24.jar
plexus-interpolation.license
plexus-sec-dispatcher-1.4.jar
plexus-sec-dispatcher.license
plexus-utils-3.1.0.jar
plexus-utils.license
slf4j-api-1.7.25.jar
slf4j-api.license
wagon-file-3.1.0.jar
wagon-file.license
wagon-http-3.1.0-shaded.jar
wagon-http.license
wagon-provider-api-3.1.0.jar
wagon-provider-api.license

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/ext:
README.txt

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native:
freebsd32
freebsd64
linux32
linux64
osx
README.txt
windows32
windows64

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/osx:
libjansi.jnilib

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows32:
jansi.dll

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows64:
jansi.dll
Finished /home/jenkins/tools/maven/apache-maven-3.5.4 Directory Listing :
Detected current version as: 
'HUDI_home=
0.5.1-SNAPSHOT'
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Hudi   [pom]
[INFO] hudi-common[jar]
[INFO] hudi-timeline-service  [jar]
[INFO] hudi-hadoop-mr [jar]
[INFO] hudi-client[jar]
[INFO] hudi-hive  [jar]
[INFO] hudi-spark [jar]
[INFO] hudi-utilities [jar]
[INFO] hudi-cli   [jar]
[INFO] hudi-hadoop-mr-bundle  [jar]
[INFO] hudi-hive-bundle   [jar]
[INFO] hudi-spark-bundle  [jar]
[INFO] hudi-presto-bundle [jar]
[INFO] hudi-utilities-bundle  [jar]
[INFO] hudi-timeline-server-bundle[jar]
[INFO] h

[GitHub] [incubator-hudi] cdmikechen edited a comment on issue #915: [HUDI-268] Shade and relocate Avro dependency in hadoop-mr-bundle

2019-09-28 Thread GitBox 
cdmikechen edited a comment on issue #915: [HUDI-268] Shade and relocate Avro 
dependency in hadoop-mr-bundle
URL: https://github.com/apache/incubator-hudi/pull/915#issuecomment-536241871
 
 
   I found that some codes does not use avro to process data structure when it 
does stream processing before. Can we refer to that part of code to convert 
avro when writing hudi data and save data with parquet's basic api?
   Now, the main problem is that some data types cannot be converted correctly 
after using avro. This problem may be solved when it is stored without avro .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] cdmikechen edited a comment on issue #915: [HUDI-268] Shade and relocate Avro dependency in hadoop-mr-bundle

2019-09-28 Thread GitBox 
cdmikechen edited a comment on issue #915: [HUDI-268] Shade and relocate Avro 
dependency in hadoop-mr-bundle
URL: https://github.com/apache/incubator-hudi/pull/915#issuecomment-536241871
 
 
   I found that some codes does not use avro to process data structure when it 
does stream processing before. Can we refer to that part of code to convert 
avro when writing hudi data and save data with parquet's basic api?
   Now, the main problem is that some data types cannot be converted correctly 
after using avro. This problem may be solved without avro when it is stored.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] cdmikechen edited a comment on issue #915: [HUDI-268] Shade and relocate Avro dependency in hadoop-mr-bundle

2019-09-28 Thread GitBox 
cdmikechen edited a comment on issue #915: [HUDI-268] Shade and relocate Avro 
dependency in hadoop-mr-bundle
URL: https://github.com/apache/incubator-hudi/pull/915#issuecomment-536241871
 
 
   I found that some codes does not use avro to process data structure when it 
does stream processing before. Can we refer to that part of code to convert 
avro when writing hudi data and save data with parquet's basic api?
   Now, the main problem is that some data types cannot be converted correctly 
after using avro. This problem may be solved without Avro when it is stored 
again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] cdmikechen commented on issue #915: [HUDI-268] Shade and relocate Avro dependency in hadoop-mr-bundle

2019-09-28 Thread GitBox 
cdmikechen commented on issue #915: [HUDI-268] Shade and relocate Avro 
dependency in hadoop-mr-bundle
URL: https://github.com/apache/incubator-hudi/pull/915#issuecomment-536241871
 
 
   I found that some codes does not use avro to process data structure when it 
does stream processing before. Can we refer to that part of code to convert 
avro when writing hudi data and save data with parquet's basic api?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HUDI-263) Windows support for Hudi writing

2019-09-28 Thread Subramanian Mohan (Jira) 


[ 
https://issues.apache.org/jira/browse/HUDI-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940159#comment-16940159
 ] 

Subramanian Mohan commented on HUDI-263:


[~vinoth] I tried at some level.  Due to some challenges on environment, I'll 
pick up later .

> Windows support for Hudi writing
> 
>
> Key: HUDI-263
> URL: https://issues.apache.org/jira/browse/HUDI-263
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Write Client
>Reporter: Vinoth Chandar
>Priority: Major
>
> https://github.com/apache/incubator-hudi/issues/905



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HUDI-263) Windows support for Hudi writing

2019-09-28 Thread Subramanian Mohan (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subramanian Mohan reassigned HUDI-263:
--

Assignee: (was: Subramanian Mohan)

> Windows support for Hudi writing
> 
>
> Key: HUDI-263
> URL: https://issues.apache.org/jira/browse/HUDI-263
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Write Client
>Reporter: Vinoth Chandar
>Priority: Major
>
> https://github.com/apache/incubator-hudi/issues/905



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bvaradar commented on a change in pull request #929: [HUDI-271] Create 
QuickstartUtils class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929#discussion_r329309194
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java
 ##
 @@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.Deflater;
+import java.util.zip.DeflaterOutputStream;
+import java.util.zip.InflaterInputStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.MercifulJsonConverter;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+/**
+ * Class to be used in quickstart guide for generating inserts and updates 
against a corpus.
+ * 
+ * Test data uses a toy Uber trips, data model.
+ *
+ */
+public class QuickstartUtils {
+
+  public static class DataGenerator {
+private static final String DEFAULT_FIRST_PARTITION_PATH = "2019/09/15";
+private static final String DEFAULT_SECOND_PARTITION_PATH = "2018/09/16";
+private static final String DEFAULT_THIRD_PARTITION_PATH = "2018/09/17";
+
+private static final String[] DEFAULT_PARTITION_PATHS = {
+DEFAULT_FIRST_PARTITION_PATH,
+DEFAULT_SECOND_PARTITION_PATH,
+DEFAULT_THIRD_PARTITION_PATH
+};
+static String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": 
\"triprec\"," + "\"fields\": [ "
++ "{\"name\": \"ts\",\"type\": \"double\"},"
++ "{\"name\": \"uuid\", \"type\": \"string\"},"
++ "{\"name\": \"rider\", \"type\": \"string\"},"
++ "{\"name\": \"driver\", \"type\": \"string\"},"
++ "{\"name\": \"begin_lat\", \"type\": \"double\"},"
++ "{\"name\": \"begin_lon\", \"type\": \"double\"},"
++ "{\"name\": \"end_lat\", \"type\": \"double\"},"
++ "{\"name\": \"end_lon\", \"type\": \"double\"},"
++ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+static Schema avroSchema = new Schema.Parser().parse(TRIP_EXAMPLE_SCHEMA);
+
+private static Random rand = new Random(46474747);
+
+private final Map existingKeys;
+private final String[] partitionPaths;
+private int numExistingKeys;
+
+public DataGenerator() {
+  this(DEFAULT_PARTITION_PATHS, new HashMap<>());
+}
+
+private DataGenerator(String[] partitionPaths, Map 
keyPartitionMap) {
+  this.partitionPaths = Arrays.copyOf(partitionPaths, 
partitionPaths.length);
+  this.existingKeys = keyPartitionMap;
+}
+
+private static String generateRandomCommitMsg() {
+  int leftLimit = 48; // ascii for 0
+  int rightLimit = 57; // ascii for 9
+  int stringLength = 3;
+  StringBuilder buffer = new StringBuilder(stringLength);
+  for (int i = 0; i < stringLength; i++) {
+int randomLimitedInt = leftLimit + (int)
+(rand.nextFloat() * (rightLimit - leftLimit + 1));
+buffer.append((char) randomLimitedInt);
+  }
+  return buffer.toString();
+}
+
+public int getNumExistingKeys() {
+  return numExistingKeys;
+}
+
+public static GenericRecord generateGenericRecord(String rowKey, String 
riderName, String driverName,
+  double timestamp) {
+  GenericRecord rec = new GenericData.Record(avroSchema);
+  rec.put

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bvaradar commented on a change in pull request #929: [HUDI-271] Create 
QuickstartUtils class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929#discussion_r329309678
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java
 ##
 @@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.Deflater;
+import java.util.zip.DeflaterOutputStream;
+import java.util.zip.InflaterInputStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.MercifulJsonConverter;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+/**
+ * Class to be used in quickstart guide for generating inserts and updates 
against a corpus.
+ * 
+ * Test data uses a toy Uber trips, data model.
+ *
+ */
+public class QuickstartUtils {
+
+  public static class DataGenerator {
+private static final String DEFAULT_FIRST_PARTITION_PATH = "2019/09/15";
+private static final String DEFAULT_SECOND_PARTITION_PATH = "2018/09/16";
+private static final String DEFAULT_THIRD_PARTITION_PATH = "2018/09/17";
+
+private static final String[] DEFAULT_PARTITION_PATHS = {
+DEFAULT_FIRST_PARTITION_PATH,
+DEFAULT_SECOND_PARTITION_PATH,
+DEFAULT_THIRD_PARTITION_PATH
+};
+static String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": 
\"triprec\"," + "\"fields\": [ "
++ "{\"name\": \"ts\",\"type\": \"double\"},"
++ "{\"name\": \"uuid\", \"type\": \"string\"},"
++ "{\"name\": \"rider\", \"type\": \"string\"},"
++ "{\"name\": \"driver\", \"type\": \"string\"},"
++ "{\"name\": \"begin_lat\", \"type\": \"double\"},"
++ "{\"name\": \"begin_lon\", \"type\": \"double\"},"
++ "{\"name\": \"end_lat\", \"type\": \"double\"},"
++ "{\"name\": \"end_lon\", \"type\": \"double\"},"
++ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+static Schema avroSchema = new Schema.Parser().parse(TRIP_EXAMPLE_SCHEMA);
+
+private static Random rand = new Random(46474747);
+
+private final Map existingKeys;
+private final String[] partitionPaths;
+private int numExistingKeys;
+
+public DataGenerator() {
+  this(DEFAULT_PARTITION_PATHS, new HashMap<>());
+}
+
+private DataGenerator(String[] partitionPaths, Map 
keyPartitionMap) {
+  this.partitionPaths = Arrays.copyOf(partitionPaths, 
partitionPaths.length);
+  this.existingKeys = keyPartitionMap;
+}
+
+private static String generateRandomCommitMsg() {
+  int leftLimit = 48; // ascii for 0
+  int rightLimit = 57; // ascii for 9
+  int stringLength = 3;
+  StringBuilder buffer = new StringBuilder(stringLength);
+  for (int i = 0; i < stringLength; i++) {
+int randomLimitedInt = leftLimit + (int)
+(rand.nextFloat() * (rightLimit - leftLimit + 1));
+buffer.append((char) randomLimitedInt);
+  }
+  return buffer.toString();
+}
+
+public int getNumExistingKeys() {
+  return numExistingKeys;
+}
+
+public static GenericRecord generateGenericRecord(String rowKey, String 
riderName, String driverName,
+  double timestamp) {
+  GenericRecord rec = new GenericData.Record(avroSchema);
+  rec.put

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bvaradar commented on a change in pull request #929: [HUDI-271] Create 
QuickstartUtils class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929#discussion_r329309323
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java
 ##
 @@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.Deflater;
+import java.util.zip.DeflaterOutputStream;
+import java.util.zip.InflaterInputStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.MercifulJsonConverter;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+/**
+ * Class to be used in quickstart guide for generating inserts and updates 
against a corpus.
+ * 
+ * Test data uses a toy Uber trips, data model.
+ *
+ */
+public class QuickstartUtils {
+
+  public static class DataGenerator {
+private static final String DEFAULT_FIRST_PARTITION_PATH = "2019/09/15";
+private static final String DEFAULT_SECOND_PARTITION_PATH = "2018/09/16";
+private static final String DEFAULT_THIRD_PARTITION_PATH = "2018/09/17";
+
+private static final String[] DEFAULT_PARTITION_PATHS = {
+DEFAULT_FIRST_PARTITION_PATH,
+DEFAULT_SECOND_PARTITION_PATH,
+DEFAULT_THIRD_PARTITION_PATH
+};
+static String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": 
\"triprec\"," + "\"fields\": [ "
++ "{\"name\": \"ts\",\"type\": \"double\"},"
++ "{\"name\": \"uuid\", \"type\": \"string\"},"
++ "{\"name\": \"rider\", \"type\": \"string\"},"
++ "{\"name\": \"driver\", \"type\": \"string\"},"
++ "{\"name\": \"begin_lat\", \"type\": \"double\"},"
++ "{\"name\": \"begin_lon\", \"type\": \"double\"},"
++ "{\"name\": \"end_lat\", \"type\": \"double\"},"
++ "{\"name\": \"end_lon\", \"type\": \"double\"},"
++ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+static Schema avroSchema = new Schema.Parser().parse(TRIP_EXAMPLE_SCHEMA);
+
+private static Random rand = new Random(46474747);
+
+private final Map existingKeys;
+private final String[] partitionPaths;
+private int numExistingKeys;
+
+public DataGenerator() {
+  this(DEFAULT_PARTITION_PATHS, new HashMap<>());
+}
+
+private DataGenerator(String[] partitionPaths, Map 
keyPartitionMap) {
+  this.partitionPaths = Arrays.copyOf(partitionPaths, 
partitionPaths.length);
+  this.existingKeys = keyPartitionMap;
+}
+
+private static String generateRandomCommitMsg() {
 
 Review comment:
   Minor: The word commit here is confusing with Hudi commit time. As we are 
not using instantTime here, can you rename it to getRandomString


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bvaradar commented on a change in pull request #929: [HUDI-271] Create 
QuickstartUtils class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929#discussion_r329309392
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java
 ##
 @@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.Deflater;
+import java.util.zip.DeflaterOutputStream;
+import java.util.zip.InflaterInputStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.MercifulJsonConverter;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+/**
+ * Class to be used in quickstart guide for generating inserts and updates 
against a corpus.
+ * 
+ * Test data uses a toy Uber trips, data model.
+ *
+ */
+public class QuickstartUtils {
+
+  public static class DataGenerator {
+private static final String DEFAULT_FIRST_PARTITION_PATH = "2019/09/15";
+private static final String DEFAULT_SECOND_PARTITION_PATH = "2018/09/16";
+private static final String DEFAULT_THIRD_PARTITION_PATH = "2018/09/17";
+
+private static final String[] DEFAULT_PARTITION_PATHS = {
+DEFAULT_FIRST_PARTITION_PATH,
+DEFAULT_SECOND_PARTITION_PATH,
+DEFAULT_THIRD_PARTITION_PATH
+};
+static String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": 
\"triprec\"," + "\"fields\": [ "
++ "{\"name\": \"ts\",\"type\": \"double\"},"
++ "{\"name\": \"uuid\", \"type\": \"string\"},"
++ "{\"name\": \"rider\", \"type\": \"string\"},"
++ "{\"name\": \"driver\", \"type\": \"string\"},"
++ "{\"name\": \"begin_lat\", \"type\": \"double\"},"
++ "{\"name\": \"begin_lon\", \"type\": \"double\"},"
++ "{\"name\": \"end_lat\", \"type\": \"double\"},"
++ "{\"name\": \"end_lon\", \"type\": \"double\"},"
++ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+static Schema avroSchema = new Schema.Parser().parse(TRIP_EXAMPLE_SCHEMA);
+
+private static Random rand = new Random(46474747);
+
+private final Map existingKeys;
+private final String[] partitionPaths;
+private int numExistingKeys;
+
+public DataGenerator() {
+  this(DEFAULT_PARTITION_PATHS, new HashMap<>());
+}
+
+private DataGenerator(String[] partitionPaths, Map 
keyPartitionMap) {
+  this.partitionPaths = Arrays.copyOf(partitionPaths, 
partitionPaths.length);
+  this.existingKeys = keyPartitionMap;
+}
+
+private static String generateRandomCommitMsg() {
+  int leftLimit = 48; // ascii for 0
+  int rightLimit = 57; // ascii for 9
+  int stringLength = 3;
+  StringBuilder buffer = new StringBuilder(stringLength);
+  for (int i = 0; i < stringLength; i++) {
+int randomLimitedInt = leftLimit + (int)
+(rand.nextFloat() * (rightLimit - leftLimit + 1));
+buffer.append((char) randomLimitedInt);
+  }
+  return buffer.toString();
+}
+
+public int getNumExistingKeys() {
+  return numExistingKeys;
+}
+
+public static GenericRecord generateGenericRecord(String rowKey, String 
riderName, String driverName,
+  double timestamp) {
+  GenericRecord rec = new GenericData.Record(avroSchema);
+  rec.put

[jira] [Updated] (HUDI-271) Simplify quickstart documentation

2019-09-28 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HUDI-271:

Labels: pull-request-available  (was: )

> Simplify quickstart documentation
> -
>
> Key: HUDI-271
> URL: https://issues.apache.org/jira/browse/HUDI-271
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Docs
>Reporter: Bhavani Sudha Saktheeswaran
>Assignee: Bhavani Sudha Saktheeswaran
>Priority: Minor
>  Labels: pull-request-available
>
> Make quickstart really simple by only using spark examples and default 
> configs for easier playing around with Hudi APIs. The intent is to introduce 
> what Hudi offers to end users as quickly as possible, without having to deal 
> with setting up Hive or other external systems. 
>  
> Help to set up Hive sync/ Hive metastore etc will be moved to other pages for 
> users who want to explore more on how to set these up after initially playing 
> around with Hudi api s.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] bhasudha opened a new pull request #929: [HUDI-271] Create QuickstartUtils class towards simplifying quickstar…

2019-09-28 Thread GitBox 
bhasudha opened a new pull request #929: [HUDI-271] Create QuickstartUtils 
class towards simplifying quickstar…
URL: https://github.com/apache/incubator-hudi/pull/929
 
 
   …t guide
   
   This will be used in Quickstart guide(Doc changes to follow in a seperate 
PR). The intention is to simplify quickstart to showcase hudi APIs by writing 
and reading using spark datasources.
   This is located in hudi-spark module intentionally to bring all the 
necessary classes in hudi-spark-bundle finally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (HUDI-262) Update Hudi website to reflect change in InputFormat Class name

2019-09-28 Thread Bhavani Sudha Saktheeswaran (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhavani Sudha Saktheeswaran resolved HUDI-262.
--
Resolution: Fixed

> Update Hudi website to reflect change in InputFormat Class name
> ---
>
> Key: HUDI-262
> URL: https://issues.apache.org/jira/browse/HUDI-262
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: asf-migration
>Reporter: Balaji Varadarajan
>Assignee: Bhavani Sudha Saktheeswaran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] yanghua edited a comment on issue #923: HUDI-247 Unify the initialization of HoodieTableMetaClient in test for hoodie-client module

2019-09-28 Thread GitBox 
yanghua edited a comment on issue #923: HUDI-247 Unify the initialization of 
HoodieTableMetaClient in test for hoodie-client module
URL: https://github.com/apache/incubator-hudi/pull/923#issuecomment-536160969
 
 
   @vinothchandar How can we get a new metaclient only through an old 
metaclient instance? IMO, we always need two parameters(`Configuration conf, 
String basePath`) to init a new meta client.
   
   I mean define a method like:
   
   ```java
   public HoodieTableMetaClient reloadMetaClient(Configuration conf, String 
basePath)
 throws IOException, DatasetNotFoundException {
   return new HoodieTableMetaClient(...);
 }
   ```
   
   Maybe I misunderstood your idea? Did you mean deeply copy an old client's 
fields into a new client?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on issue #923: HUDI-247 Unify the initialization of HoodieTableMetaClient in test for hoodie-client module

2019-09-28 Thread GitBox 
yanghua commented on issue #923: HUDI-247 Unify the initialization of 
HoodieTableMetaClient in test for hoodie-client module
URL: https://github.com/apache/incubator-hudi/pull/923#issuecomment-536160969
 
 
   @vinothchandar How can we get a new metaclient only through an old 
metaclient instance? IMO, we always need two parameters(`Configuration conf, 
String basePath`) to init a new meta client.
   
   I mean define a method like:
   
   ```
   public HoodieTableMetaClient reloadMetaClient(Configuration conf, String 
basePath)
 throws IOException, DatasetNotFoundException {
   return new HoodieTableMetaClient(...);
 }
   ```
   
   Maybe I misunderstood your idea? Did you mean deeply copy an old client's 
fields into a new client?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services