Are you querying a new storage plugin (I saw the stack trace shows
"indexR"), and that storage plugin is using
HardAffinityFragmentParallelizer? As far as I know, the hard assignment is
only used for distributed system tables, or some special operator (Screen)
in Drill codebase. The majority cases use soft one. Hard assignment means
to assign at least one minor fragment to each available drill endpoint.
For instance, screen has to assign to the foreman node, while distributed
system tables requires at least one minor fragment for each drillbit.
In your case, seems you have two drillbits, while the fragment # is only 1.
That fails the requirement for hard assignment. Either you choose to use
soft assignment, or make # of drillbits <= # of fragments.
On Wed, Aug 2, 2017 at 1:17 AM, 何建军 <hejianjun...@163.com> wrote:
> Hi , nice to meet you .
> thinks to read this email.
> when used drill to Query, meets some error very confused me, and i don't
> know how to solve it ,hope you could check and help me thinks a lot .
>
>
> to find any useful message ,so i modify some class to print error.
> here is my modify :
> drill-java-exec-1.9.0.jar
> org.apache.drill.exec.planner.fragment.HardAffinityFragmentParalleliz
> er.java
> line 80
> from
> checkOrThrow(endpointPool.size() <= width, logger,
> "Number of mandatory endpoints ({}) that require an assignment is
> more than the allowed fragment max " +
> "width ({}).", endpointPool.size(), pInfo.getMaxWidth());
> to
> checkOrThrow(endpointPool.size() <= width, logger,
> "Number of mandatory endpoints ({}) that require an assignment
> is more than the allowed fragment max " +
> "width ({}), fragment: {}, endpointPool: {},
> pInfo.getEndpointAffinityMap(): {}.",
> endpointPool.size(),
> pInfo.getMaxWidth(),
> fragmentWrapper.getNode(),
> org.apache.commons.lang3.StringUtils.join(
> endpointPool.values(), ","),
> org.apache.commons.lang3.StringUtils.join(pInfo.
> getEndpointAffinityMap().values(), ","));
>
>
> here is the meesage of error log
>
>
>
>
> [Error Id: 258f549e-d618-4dcb-994a-3e3908346033 on slave1:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> PhysicalOperatorSetupException: Can not parallelize fragment.
>
>
>
>
> [Error Id: 258f549e-d618-4dcb-994a-3e3908346033 on slave1:31010]
> at org.apache.drill.common.exceptions.UserException$
> Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:825)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:935)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected exception during fragment initialization: Can not parallelize
> fragment.
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:282)
> [drill-java-exec-1.9.0.jar:1.9.0]
> ... 3 common frames omitted
> Caused by: org.apache.drill.exec.physical.PhysicalOperatorSetupException:
> Can not parallelize fragment.
> at org.apache.drill.exec.planner.fragment.HardAffinityFragmentParalleliz
> er.checkOrThrow(HardAffinityFragmentParallelizer.java:162)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.planner.fragment.HardAffinityFragmentParalleliz
> er.parallelizeFragment(HardAffinityFragmentParallelizer.java:89)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.planner.fragment.SimpleParallelizer.
> parallelizeFragment(SimpleParallelizer.java:253)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.planner.fragment.SimpleParallelizer.
> getFragmentsHelper(SimpleParallelizer.java:167)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.planner.fragment.SimpleParallelizer.getFragments(SimpleParallelizer.java:126)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.
> getQueryWorkUnit(Foreman.java:596) [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.
> runPhysicalPlan(Foreman.java:426) [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1010)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
> [drill-java-exec-1.9.0.jar:1.9.0]
> ... 3 common frames omitted
> 2017-07-27 12:01:15,912 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 26869a74-53da-2307-1e8d-9326be0d04eb: select sum(t.num) num from (select
> province,count(distinct phone) as num from indexr.count_phone where ctime
> <= 20170725 group by province) t
> 2017-07-27 12:01:16,100 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 6 classes for
> org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
> 2017-07-27 12:01:16,101 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 7 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-07-27 12:01:16,101 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 7 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-07-27 12:01:16,101 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 7 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-07-27 12:01:16,117 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 0hjj=======================maxPw=:
> 1
> 2017-07-27 12:01:16,117 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 1hjj=======================maxPw=:
> 101
> 2017-07-27 12:01:16,139 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 0hjj=======================maxPw=:
> 1
> 2017-07-27 12:01:16,139 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 1hjj=======================maxPw=:
> 101
> 2017-07-27 12:01:16,142 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 0hjj=======================maxPw=:
> 1
> 2017-07-27 12:01:16,142 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 1hjj=======================maxPw=:
> 101
> 2017-07-27 12:01:16,153 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 33hjj=======================maxPw=:
> 1
> 2017-07-27 12:01:16,153 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> INFO o.a.d.e.s.indexr.ScanWrokProvider - 44hjj=======================maxPw=:
> 1001
> 2017-07-27 12:01:16,154 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> ERROR o.a.d.e.p.f.HardAffinityFragmentParallelizer -
> 000000============width=2
> 2017-07-27 12:01:16,154 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> ERROR o.a.d.e.p.f.HardAffinityFragmentParallelizer -
> 111111============width=1 pInfo.getMaxWidth()=1
> 2017-07-27 12:01:16,154 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> ERROR o.a.d.e.p.f.HardAffinityFragmentParallelizer - Number of mandatory
> endpoints (2) that require an assignment is more than the allowed fragment
> max width (1), fragment: FragmentNode [root=Screen [endpoint=address:
> "slave1"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> , getChild()=org.apache.drill.exec.physical.config.Project@ae9bd17],
> sendingExchange=null, receivingExchangePairs=[]], endpointPool:
> EndpointAffinity [endpoint=address: "slave1" user_port: 31010 control_port:
> 31011 data_port: 31012, affinity=1.0, mandatory=true,
> maxWidth=1],EndpointAffinity [endpoint=address: "slave2" user_port: 31010
> control_port: 31011 data_port: 31012, affinity=3616256.0, mandatory=true,
> maxWidth=2147483647], pInfo.getEndpointAffinityMap(): EndpointAffinity
> [endpoint=address: "slave1" user_port: 31010 control_port: 31011 data_port:
> 31012, affinity=1.0, mandatory=true, maxWidth=1],EndpointAffinity
> [endpoint=address: "slave2" user_port: 31010 control_port: 31011 data_port:
> 31012, affinity=3616256.0, mandatory=true, maxWidth=2147483647].
> 2017-07-27 12:01:16,160 [26869a74-53da-2307-1e8d-9326be0d04eb:foreman]
> ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR:
> PhysicalOperatorSetupException: Can not parallelize fragment.
>
>
>
>
> [Error Id: 7d2c1de5-83c7-473e-aba4-8540c39e14d8 on slave1:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> PhysicalOperatorSetupException: Can not parallelize fragment.
>
>
>
>
> [Error Id: 7d2c1de5-83c7-473e-aba4-8540c39e14d8 on slave1:31010]
> at org.apache.drill.common.exceptions.UserException$
> Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:825)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:935)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected exception during fragment initialization: Can not parallelize
> fragment.
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:282)
> [drill-java-exec-1.9.0.jar:1.9.0]
> ... 3 common frames omitted
> Caused by: org.apache.drill.exec.physical.PhysicalOperatorSetupException:
> Can not parallelize fragment.
> at org.apache.drill.exec.planner.fragment.HardAffinityFragmentParalleliz
> er.checkOrThrow(HardAffinityFragmentParallelizer.java:162)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.planner.fragment.HardAffinityFragmentParalleliz
> er.parallelizeFragment(HardAffinityFragmentParallelizer.java:89)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.planner.fragment.SimpleParallelizer.
> parallelizeFragment(SimpleParallelizer.java:253)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.planner.fragment.SimpleParallelizer.
> getFragmentsHelper(SimpleParallelizer.java:167)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.planner.fragment.SimpleParallelizer.getFragments(SimpleParallelizer.java:126)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.
> getQueryWorkUnit(Foreman.java:596) [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.
> runPhysicalPlan(Foreman.java:426) [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1010)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
> [drill-java-exec-1.9.0.jar:1.9.0]
> ... 3 common frames omitted
>
>
>
>
