on.sort.size=5 ( by default is 50, please set to 1/10 for
>reducing temp intermediate files)
>
>
>Regards
>Liang
>
>
>
>--
>View this message in context:
>http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/insert-into-carbon-table-failed-t
Hi
Please enable vector , it might help limit query.
import org.apache.carbondata.core.util.CarbonProperties
import org.apache.carbondata.core.constants.CarbonCommonConstants
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_VECTOR_READER,
"true")
Regards
Liang
a wrote
>
I guess then word node in "Carbodata launches one job per each node to sort
the data at node level and
avoid shuffling" may make some confuse. I guess carbondata should launches
one task per each executor . here job should be task ,node should be
executor.
Maybe he can try increase the number of
, please set to 1/10 for
reducing temp intermediate files)
Regards
Liang
--
View this message in context:
http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/insert-into-carbon-table-failed-tp9609p9688.html
Sent from the Apache CarbonData Mailing List archive mailing list archive
TEST SQL ：
高基数随机查询
select * From carbon_table where dt='2017-01-01' and user_id='' limit 100;
高基数随机查询like
select * From carbon_table where dt='2017-01-01' and fo like '%%' limit 100;
低基数随机查询
select * From carbon_table where dt='2017-01-01' and plat='android' and
tv='8400' limit 100
1
Hi,
It is little weird, I tried to reproduce this issue but I am not
successful. Can you make sure that latest jar is updated in all the
datanodes and driver. There may be possibility that old jar is still
referring in either driver or in datanode.
Regards,
Ravindra
On 27 March 2017 at 01:40, a
I download the newest sourcecode (master) and compile,generate the jar
carbondata_2.11-1.1.0-incubating-SNAPSHOT-shade-hadoop2.7.2.jar
Then i use spark2.1 test again.The error logs are as follow:
Container log :
17/03/27 02:27:21 ERROR newflow.DataLoadExecutor: Executor task launch worker-9
D
Hi,
Please try to run on the master branch. As I mentioned earlier there are
few memory issues in 1.0 release. We already initiated new release 1.1.0,
so better try to run on the latest code.
And also please make sure that property
enable.unsafe.sort=true available to all nodes. It means carbon.
Container log : error executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL
15: SIGTERM。
spark log: 17/03/26 23:40:30 ERROR YarnScheduler: Lost executor 2 on hd25:
Container killed by YARN for exceeding memory limits. 49.0 GB of 49 GB physical
memory used. Consider boosting spark.yarn.exec
I have set the parameters as follow：
1、fs.hdfs.impl.disable.cache=true
2、dfs.socket.timeout=180 （Exception：aused by: java.io.IOException:
Filesystem closed）
3、dfs.datanode.socket.write.timeout=360
4、set carbondata property enable.unsafe.sort=true
5、remove BUCKETCOLUMNS property from the
Thank you Ravindra!
Version:
My carbondata version is 1.0,spark version is 1.6.3,hadoop version is
2.7.1,hive version is 1.1.0
one of the containers log:
17/03/25 22:07:09 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL
15: SIGTERM
17/03/25 22:07:09 INFO storage.DiskBlockManager: Sh
Hi,
Carbodata launches one job per each node to sort the data at node level and
avoid shuffling. Internally it uses threads to use parallel load. Please
use carbon.number.of.cores.while.loading property in carbon.properties file
and set the number of cores it should use per machine while loading.
jobs.How to use this property?
>> ExampleUtils :
>> // whether use table split partition
>> // true -> use table split partition, support multiple partition
>> loading
>> // false -> use node split partition, support data load by host
>> partition
>>
>> CarbonProperties.getInstance().addProperty("carbon.table.split.partition.enable",
>> "false")
>> 4、Insert into carbon table takes 3 hours ,but eventually failed 。How can i
>> speed it.
>> 5、in the spark-shell ,I tried to set this parameter range from 10 to
>> 20,but the second job has only 5 tasks
>> the other parameter executor-memory = 20G is enough?
>>
>> I need your help!Thank you very much!
>
>> wwyxg@
>
>>
>>
>
>> wwyxg@
>
>
>
>
>
>--
>View this message in context:
>http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/insert-into-carbon-table-failed-tp9609p9610.html
>Sent from the Apache CarbonData Mailing List archive mailing list archive at
>Nabble.com.
gt;> I set "carbon.table.split.partition.enable" to true, but the second
>> job has only five jobs.How to use this property?
>> ExampleUtils :
>> // whether use table split partition
>> // true -> use table split partition, support multiple partition
gt;
> CarbonProperties.getInstance().addProperty("carbon.table.split.partition.enable",
> "false")
> 4、Insert into carbon table takes 3 hours ,but eventually failed 。How can i
> speed it.
> 5、in the spark-shell ,I tried to set this parameter range from 10 to
> 20,but the second job has only 5 tasks
> the other parameter executor-memory = 20G is enough?
>
> I need your help!Thank you very much!
> wwyxg@
>
>
> wwyxg@
--
View this message in context:
http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/insert-into-carbon-table-failed-tp9609p9610.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at
Nabble.com.
Hello!
0、The failure
When i insert into carbon table，i encounter failure。The failure is as follow:
Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most
recent failure: Lost task 0.3 in stage 2.0 (TID 1007, hd26):
ExecutorLostFailure (executor 1 exited caused by one of the
16 matches
Mail list logo
