Hi liyinwei
Very good! You are the person who i met learnt Apache CarbonData fastest!
Can you raise one mailing list discussion about improving log info what you
mentioned.
Look forward to seeing your code contribution :)
Regards
Liang
2017-01-10 14:44 GMT+08:00 251469031 <251469...@qq.com>:
> thx 陈亮总。
>
>
> I've solved the problem, here is my record:
>
>
> first,
>
>
> I found the spark job failed when loading data and there is an error
> "CarbonDataWriterException: Problem while copying file from local store to
> carbon store", when located to the source code at
> ./processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter,
> it shows:
>
>
> private void copyCarbonDataFileToCarbonStorePath(String localFileName)
> throws CarbonDataWriterException {
> long copyStartTime = System.currentTimeMillis();
> LOGGER.info("Copying " + localFileName + " --> " + dataWriterVo.
> getCarbonDataDirectoryPath());
> try {
> CarbonFile localCarbonFile =
> FileFactory.getCarbonFile(localFileName,
> FileFactory.getFileType(localFileName));
> String carbonFilePath = dataWriterVo.getCarbonDataDirectoryPath() +
> localFileName
> .substring(localFileName.lastIndexOf(File.separator));
> copyLocalFileToCarbonStore(carbonFilePath, localFileName,
> CarbonCommonConstants.BYTEBUFFER_SIZE,
> getMaxOfBlockAndFileSize(fileSizeInBytes,
> localCarbonFile.getSize()));
> } catch (IOException e) {
> throw new CarbonDataWriterException(
> "Problem while copying file from local store to carbon store");
> }
> LOGGER.info(
> "Total copy time (ms) to copy file " + localFileName + " is " +
> (System.currentTimeMillis()
> - copyStartTime));
> }
>
>
>
> the main reason is that the method copyLocalFileToCarbonStore cause an
> IOException, but the catch block doesn't tell me what is the real reason
> that coused the error(at this moment, I really like technical logs more
> then business logs). so I add a line of code:
> ...
> catch (IOException e) {
> LOGGER.info("-------------------logs print by liyinwei
> start---------------------");
> LOGGER.error(e, "");
> LOGGER.info("-------------------logs print by liyinwei end
> ---------------------");
> throw new CarbonDataWriterException(
> "Problem while copying file from local store to carbon store");
>
>
>
> then I rebuild the source code and it logs as follows:
>
>
> INFO 10-01 10:29:59,546 - [test_table: Graph -
> MDKeyGentest_table][partitionID:0]
> -------------------logs print by liyinwei start---------------------
> ERROR 10-01 10:29:59,547 - [test_table: Graph - MDKeyGentest_table][
> partitionID:0]
> java.io.FileNotFoundException: /home/hadoop/carbondata/bin/
> carbonshellstore/default/test_table/Fact/Part0/Segment_0/
> part-0-0-1484015398000.carbondata (No such file or directory)
> at java.io.FileOutputStream.open0(Native Method)
> ...
> INFO 10-01 10:29:59,547 - [test_table: Graph -
> MDKeyGentest_table][partitionID:0]
> -------------------logs print by liyinwei end ---------------------
> ERROR 10-01 10:29:59,547 - [test_table: Graph -
> MDKeyGentest_table][partitionID:0]
> Problem while copying file from local store to carbon store
>
>
>
> second,
>
>
> as u see, the main reason that cause the error is a FileNotFoundException,
> which means the metadata is not found. with the help of Liang Chen & Brave
> heart, I found that the default of carbondata storePath is as below if we
> start the spark-shell by using carbon-spark-shell:
> scala> print(cc.storePath)
> /home/hadoop/carbondata/bin/carbonshellstore
>
>
>
> so I added a parameter when starting carbon-spark-shell:
> ./bin/carbon-spark-shell --conf spark.carbon.storepath=hdfs://
> master:9000/home/hadoop/carbondata/bin/carbonshellstore
>
>
> and then print the storePath:
> scala> print(cc.storePath)
> hdfs://master:9000/home/hadoop/carbondata/bin/carbonshellstore
>
>
>
>
>
> finally,
>
>
> I run the command
>
>
> cc.sql(s"load data inpath 'hdfs://master:9000/home/hadoop/sample.csv'
> into table test_table")
>
>
> again and it success, which follows:
>
>
> cc.sql("select * from test_table").show
>
>
>
>
>
>
> ------------------ Original ------------------
> From: "Liang Chen";<chenliang6...@gmail.com>;
> Date: Tue, Jan 10, 2017 12:11 PM
> To: "dev"<dev@carbondata.incubator.apache.org>;
>
> Subject: Re: Problem while copying file from local store to carbon store
>
>
>
> Hi
>
> Please use spark-shell to create carboncontext, you can refer to these
> articles :
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67635497
>
> Regards
> Liang
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/load-data-error-
> from-csv-file-at-hdfs-error-in-standalone-spark-cluster-tp5783p5844.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
--
Regards
Liang
