[
https://issues.apache.org/jira/browse/KYLIN-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088397#comment-17088397
]
ASF GitHub Bot commented on KYLIN-4320:
---------------------------------------
coveralls commented on issue #1180:
URL: https://github.com/apache/kylin/pull/1180#issuecomment-617017531
## Pull Request Test Coverage Report for [Build
5837](https://coveralls.io/builds/30217120)
* **0** of **0** changed or added relevant lines in **0** files are
covered.
* **1** unchanged line in **1** file lost coverage.
* Overall coverage increased (+**0.0003%**) to **27.466%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
|
[server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java](https://coveralls.io/builds/30217120/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Futil%2FQueryRequestLimits.java#L72)
| 1 | 47.62% |
<!-- | **Total:** | **1** | | -->
| Totals | [](https://coveralls.io/builds/30217120)
|
| :-- | --: |
| Change from base [Build 5830](https://coveralls.io/builds/30067502): |
0.0003% |
| Covered Lines: | 24316 |
| Relevant Lines: | 88532 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> number of replicas of Cuboid files cannot be configured for Spark engine
> ------------------------------------------------------------------------
>
> Key: KYLIN-4320
> URL: https://issues.apache.org/jira/browse/KYLIN-4320
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v3.0.1
> Reporter: Congling Xia
> Assignee: Yaqian Zhang
> Priority: Major
> Fix For: v3.1.0
>
> Attachments: cuboid_replications.png
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Hi, team. I try to change `dfs.replication` to 3 by adding the following
> config override
> {code:java}
> kylin.engine.spark-conf.spark.hadoop.dfs.replication=3
> {code}
> Then, I get a strange result - numbers of replicas of cuboid files varies
> even though they are in the same level.
> !cuboid_replications.png!
> I guess it is due to the conflicting settings in SparkUtil:
> {code:java}
> public static void modifySparkHadoopConfiguration(SparkContext sc) throws
> Exception {
> sc.hadoopConfiguration().set("dfs.replication", "2"); // cuboid
> intermediate files, replication=2
>
> sc.hadoopConfiguration().set("mapreduce.output.fileoutputformat.compress",
> "true");
>
> sc.hadoopConfiguration().set("mapreduce.output.fileoutputformat.compress.type",
> "BLOCK");
>
> sc.hadoopConfiguration().set("mapreduce.output.fileoutputformat.compress.codec",
> "org.apache.hadoop.io.compress.DefaultCodec"); // or
> org.apache.hadoop.io.compress.SnappyCodec
> }
> {code}
> It may be a bug for Spark property precedence. After checking [Spark
> documents|#dynamically-loading-spark-properties]], it seems that some
> programmatically set properties may not take effect and it is not a
> recommended way for Spark job configuration.
> Â
> Anyway, cuboid files may survive for weeks until expired or been merged, the
> configuration rewrite in
> `org.apache.kylin.engine.spark.SparkUtil#modifySparkHadoopConfiguration`
> makes those files less reliable.
> Is there any way to force cuboid files to remain 3 replicas? or shall we
> remove the code in SparkUtil to make
> kylin.engine.spark-conf.spark.hadoop.dfs.replication work properly?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
