[jira] [Updated] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-6906:
Fix Version/s: (was: 1.15.0)
1.16.0
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.16.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
> security.auth.keytab : "/etc/drill/mapr_maprsasl.keytab",
> security.user.auth.enabled: true,
> security.user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> security.user.auth.impl: "pam4j",
> security.user.auth.pam_profiles: ["sudo", "login"],
> http.ssl_enabled: true,
> ssl.useHadoopConfig: true,
> http.auth.mechanisms: ["FORM", "SPNEGO"],
> http.auth.spnego.principal: "HTTP/perfnode206.perf@qa.lab",
> http.auth.spnego.keytab: "/etc/drill_spnego/perfnode206.keytab"
> }
> {noformat}
--
[jira] [Commented] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723635#comment-16723635
]
Sorabh Hamirwasia commented on DRILL-6906:
--
This was found to be the config issue on MapR cluster where below variable was
expected to be set for impersonation to work:
{code:java}
export MAPR_IMPERSONATION_ENABLED=${MAPR_IMPERSONATION_ENABLED:-"true"}
{code}
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
> security.auth.keytab : "/etc/drill/mapr_maprsasl.keytab",
> security.user.auth.enabled: true,
> security.user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> security.user.auth.impl: "pam4j",
> security.user.auth.pam_profiles: ["sudo", "login"],
> http.ssl_enabled: true,
> ssl.useHadoopConfig: true,
[jira] [Comment Edited] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723635#comment-16723635
]
Sorabh Hamirwasia edited comment on DRILL-6906 at 12/18/18 3:44 AM:
This was found to be the config issue on MapR cluster where below variable was
expected to be set in drill-env.sh for impersonation to work:
{code:java}
export MAPR_IMPERSONATION_ENABLED=${MAPR_IMPERSONATION_ENABLED:-"true"}
{code}
was (Author: shamirwasia):
This was found to be the config issue on MapR cluster where below variable was
expected to be set for impersonation to work:
{code:java}
export MAPR_IMPERSONATION_ENABLED=${MAPR_IMPERSONATION_ENABLED:-"true"}
{code}
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
>
[jira] [Assigned] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia reassigned DRILL-6906:
Assignee: Kunal Khatua (was: Pritesh Maker)
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
> security.auth.keytab : "/etc/drill/mapr_maprsasl.keytab",
> security.user.auth.enabled: true,
> security.user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> security.user.auth.impl: "pam4j",
> security.user.auth.pam_profiles: ["sudo", "login"],
> http.ssl_enabled: true,
> ssl.useHadoopConfig: true,
> http.auth.mechanisms: ["FORM", "SPNEGO"],
> http.auth.spnego.principal: "HTTP/perfnode206.perf@qa.lab",
> http.auth.spnego.keytab: "/etc/drill_spnego/perfnode206.keytab"
> }
> {noformat}
--
[jira] [Updated] (DRILL-6906) File permissions are not being honored
[
https://issues.apache.org/jira/browse/DRILL-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6906:
-
Priority: Major (was: Blocker)
> File permissions are not being honored
> --
>
> Key: DRILL-6906
> URL: https://issues.apache.org/jira/browse/DRILL-6906
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Client - ODBC
>Affects Versions: 1.15.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
>Priority: Major
> Fix For: 1.15.0
>
>
> I ran sqlline with user "kuser1".
> {noformat}
> /opt/mapr/drill/drill-1.15.0.apache/bin/sqlline -u
> "jdbc:drill:drillbit=10.10.30.206" -n kuser1 -p mapr
> {noformat}
> I tried to access a file that is only accessible by root:
> {noformat}
> [root@perfnode206 drill-test-framework_krystal]# hf -ls
> /drill/testdata/impersonation/neg_tc5/student
> -rwx-- 3 root root 64612 2018-06-19 10:30
> /drill/testdata/impersonation/neg_tc5/student
> {noformat}
> I am able to read the table, which should not be possible. I used this
> commit for Drill 1.15.
> {noformat}
> git.commit.id=bf2b414ac62cfc515fdd77f2688bb110073d764d
> git.commit.message.full=DRILL-6866\: Upgrade to SqlLine 1.6.0\n\n1. Changed
> SqlLine version to 1.6.0.\n2. Overridden new getVersion method in
> DrillSqlLineApplication.\n3. Set maxColumnWidth to 80 to avoid issue
> described in DRILL-6769.\n4. Changed colorScheme to obsidian.\n5. Output null
> value for varchar / char / boolean types as null instead of empty string.\n6.
> Changed access modifier from package default to public for JDBC classes that
> implement external interfaces to avoid issues when calling methods from these
> classes using reflection.\n\ncloses \#1556
> {noformat}
> This is from drillbit.log. It shows that user is kuser1.
> {noformat}
> 2018-12-15 05:00:52,516 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] DEBUG
> o.a.d.e.w.f.QueryStateProcessor - 23eb04fb-1701-bea7-dd97-ecda58795b3b: State
> change requested PREPARING --> PLANNING
> 2018-12-15 05:00:52,531 [23eb04fb-1701-bea7-dd97-ecda58795b3b:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23eb04fb-1701-bea7-dd97-ecda58795b3b issued by kuser1: select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student`
> {noformat}
> It is not clear to me if this is a Drill problem or a file system problem. I
> tested MFS by logging in as kuser1 and trying to copy the file using "hadoop
> fs -copyToLocal /drill/testdata/impersonation/neg_tc5/student" and got an
> error, and was not able to copy the file. So I think MFS permissions are
> working.
> I also tried with Drill 1.14, and I get the expected error:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.30.206> select * from
> dfs.`/drill/testdata/impersonation/neg_tc5/student` limit 1;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
> '/drill/testdata/impersonation/neg_tc5/student' not found within 'dfs'
> [Error Id: cdf18c2a-b005-4f92-b819-d4324e8807d9 on
> perfnode206.perf.lab:31010] (state=,code=0)
> {noformat}
> The commit for Drill 1.14 is:
> {noformat}
> git.commit.message.full=[maven-release-plugin] prepare release drill-1.14.0\n
> git.commit.id=0508a128853ce796ca7e99e13008e49442f83147
> {noformat}
> This problem exists with both Apache JDBC and Simba ODBC.
> Here is drill-distrib.conf. drill-override.conf is empty. It is the same
> for both 1.14 and 1.15.
> {noformat}
> drill.exec: {
> cluster-id: "secure206-drillbits",
> zk.connect:
> "perfnode206.perf.lab:5181,perfnode207.perf.lab:5181,perfnode208.perf.lab:5181",
> rpc.user.client.threads: "4",
> options.store.parquet.block-size: "268435456",
> sys.store.provider.zk.blobroot: "maprfs:///apps/drill",
> spill.directories: [ "/tmp/drill/spill" ],
> spill.fs: "maprfs:///",
> storage.action_on_plugins_override_file: "rename"
> zk.apply_secure_acl: true,
> impersonation.enabled: true,
> impersonation.max_chained_user_hops: 3,
> options.exec.impersonation.inbound_policies:
> "[{proxy_principals:{users:[\"mapr\"]},target_principals:{users:[\"*\"]}}]",
> security.auth.mechanisms: ["PLAIN", "KERBEROS"],
> security.auth.principal : "mapr/maprs...@qa.lab",
> security.auth.keytab : "/etc/drill/mapr_maprsasl.keytab",
> security.user.auth.enabled: true,
> security.user.auth.packages += "org.apache.drill.exec.rpc.user.security",
> security.user.auth.impl: "pam4j",
> security.user.auth.pam_profiles: ["sudo", "login"],
> http.ssl_enabled: true,
> ssl.useHadoopConfig: true,
> http.auth.mechanisms: ["FORM", "SPNEGO"],
> http.auth.spnego.principal: "HTTP/perfnode206.perf@qa.lab",
> http.auth.spnego.keytab: "/etc/drill_spnego/perfnode206.keytab"
> }
> {noformat}
--
This message was
[jira] [Updated] (DRILL-6879) Indicate a warning in the WebUI when a query makes little to no progress for a while
[
https://issues.apache.org/jira/browse/DRILL-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-6879:
Labels: doc-impacting user-experience (was: user-experience)
> Indicate a warning in the WebUI when a query makes little to no progress for
> a while
>
>
> Key: DRILL-6879
> URL: https://issues.apache.org/jira/browse/DRILL-6879
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Monitoring, Web Server
>Affects Versions: 1.14.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, user-experience
> Fix For: 1.16.0
>
> Attachments: image-2018-12-04-11-54-54-247.png,
> image-2018-12-06-11-19-00-339.png, image-2018-12-06-11-27-14-719.png
>
>
> When running a very large query on a cluster with limited resource, we
> noticed that one of the node's VM thread freezes the fragment threads as it
> tries to do some work (GC perhaps?). This is a clear indication that the
> query is stuck in a weird state where it might not recover from.
> Under such circumstances, it makes sense to cancel or atleast warn the user
> on that page of the query exceeding a certain threshold.
> For detecting this, the user will find that the {{Last Progress}} column in
> the Fragments Overview section will show large times.
> !image-2018-12-04-11-54-54-247.png|width=969,height=336!
> In addition, there are instances where a query might have buffered operators
> spilling to disk, which also hits performance (and, subsequently, longer run
> times). Calling out this skew can be very useful.
> !image-2018-12-06-11-27-14-719.png|width=969,height=256!
> Or there might be cases where a single fragment takes much longer than the
> average (indicated by an extreme skew in the Gantt chart).
> !image-2018-12-06-11-19-00-339.png|width=969,height=150!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-1328) Support table statistics
[ https://issues.apache.org/jira/browse/DRILL-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-1328: - Fix Version/s: (was: Future) 1.16.0 > Support table statistics > > > Key: DRILL-1328 > URL: https://issues.apache.org/jira/browse/DRILL-1328 > Project: Apache Drill > Issue Type: Improvement >Reporter: Cliff Buchanan >Assignee: Gautam Parai >Priority: Major > Fix For: 1.16.0 > > Attachments: 0001-PRE-Set-value-count-in-splitAndTransfer.patch > > > This consists of several subtasks > * implement operators to generate statistics > * add "analyze table" support to parser/planner > * create a metadata provider to allow statistics to be used by optiq in > planning optimization > * implement statistics functions > Right now, the bulk of this functionality is implemented, but it hasn't been > rigorously tested and needs to have some definite answers for some of the > parts "around the edges" (how analyze table figures out where the table > statistics are located, how a table "append" should work in a read only file > system) > Also, here are a few known caveats: > * table statistics are collected by creating a sql query based on the string > path of the table. This should probably be done with a Table reference. > * Case sensitivity for column statistics is probably iffy > * Math for combining two column NDVs into a joint NDV should be checked. > * Schema changes aren't really being considered yet. > * adding getDrillTable is probably unnecessary; it might be better to do > getTable().unwrap(DrillTable.class) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6908) Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile()
[ https://issues.apache.org/jira/browse/DRILL-6908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Guzenko closed DRILL-6908. --- Resolution: Won't Fix CSVTestBuilder is not suitable for tests with big variety of columns datatypes and it would be worthless to spend a lot of time for fixing the cast issues related to the test builder's validation query. > Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile() > -- > > Key: DRILL-6908 > URL: https://issues.apache.org/jira/browse/DRILL-6908 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Igor Guzenko >Priority: Minor > > For improving Hive tests readability it would be nice to rewrite some > cumbersome tests to use CSVTestBuilder for assertions. This issue was created > because it requires a lot of changes for different Hive tests and also some > additional time may be necessary to ensure that CSVTestBuilder can read all > datatypes (dates with different formats, floating point numbers, big decimals > etc.) into correct values for assertions. > Below is list of test methods to be rewritten: > * TestHiveStorage.readAllSupportedHiveDataTypes() > * TestHiveStorage.orderByOnHiveTable() > * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() > > * TestHiveViewsSupport.selectStarFromView() > * TestHiveViewsSupport.useHiveAndSelectStarFromView() > * TestHiveViewsSupport.viewWithAllSupportedDataTypes() > * > TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6909) Graceful shutdown does not kill the process when drill is run in embedded mode
[ https://issues.apache.org/jira/browse/DRILL-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-6909: --- Issue Type: Sub-task (was: Bug) Parent: DRILL-6023 > Graceful shutdown does not kill the process when drill is run in embedded mode > -- > > Key: DRILL-6909 > URL: https://issues.apache.org/jira/browse/DRILL-6909 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.13.0 >Reporter: Volodymyr Vysotskyi >Assignee: Venkata Jyothsna Donapati >Priority: Minor > > Graceful shutdown does not kill the process when drill is run in embedded > mode. > Steps to reproduce: > 1. Run drill in embedded mode. > 2. Press the "Shutdown" button in Web-UI. > 3. Check that the process which corresponds to the Drill wasn't killed. > This issue is observed only for embedded mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6909) Graceful shutdown does not kill the process when drill is run in embedded mode
Volodymyr Vysotskyi created DRILL-6909: -- Summary: Graceful shutdown does not kill the process when drill is run in embedded mode Key: DRILL-6909 URL: https://issues.apache.org/jira/browse/DRILL-6909 Project: Apache Drill Issue Type: Bug Affects Versions: 1.13.0 Reporter: Volodymyr Vysotskyi Assignee: Venkata Jyothsna Donapati Graceful shutdown does not kill the process when drill is run in embedded mode. Steps to reproduce: 1. Run drill in embedded mode. 2. Press the "Shutdown" button in Web-UI. 3. Check that the process which corresponds to the Drill wasn't killed. This issue is observed only for embedded mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6908) Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile()
[ https://issues.apache.org/jira/browse/DRILL-6908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6908: Description: For improving Hive tests readability it would be nice to rewrite some cumbersome tests to use CSVTestBuilder for assertions. This issue was created because it requires a lot of changes for different Hive tests and also some additional time may be necessary to ensure that CSVTestBuilder can read all datatypes (dates with different formats, floating point numbers, big decimals etc.) into correct values for assertions. Below is list of test methods to be rewritten: * TestHiveStorage.readAllSupportedHiveDataTypes() * TestHiveStorage.orderByOnHiveTable() * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() * TestHiveViewsSupport.selectStarFromView() * TestHiveViewsSupport.useHiveAndSelectStarFromView() * TestHiveViewsSupport.viewWithAllSupportedDataTypes() * TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() was: For improving Hive tests readability it would be nice to rewrite some cumbersome tests to use CSVTestBuilder for assertions. This issue was created because it requires a lot of changes for different Hive tests and also some additional time may be necessary to ensure that CSVTestBuilder can read all datatypes (dates with different formats, floating point numbers, big decimals etc.) into correct values for assertions. Below is list of test methods to be rewritten: * TestHiveStorage.readAllSupportedHiveDataTypes() * TestHiveStorage.orderByOnHiveTable() * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() * TestHiveViewsSupport.selectStarFromView() * TestHiveViewsSupport.useHiveAndSelectStarFromView() * TestHiveViewsSupport.viewWithAllSupportedDataTypes() * TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() > Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile() > -- > > Key: DRILL-6908 > URL: https://issues.apache.org/jira/browse/DRILL-6908 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Igor Guzenko >Priority: Minor > > For improving Hive tests readability it would be nice to rewrite some > cumbersome tests to use CSVTestBuilder for assertions. This issue was created > because it requires a lot of changes for different Hive tests and also some > additional time may be necessary to ensure that CSVTestBuilder can read all > datatypes (dates with different formats, floating point numbers, big decimals > etc.) into correct values for assertions. > Below is list of test methods to be rewritten: > * TestHiveStorage.readAllSupportedHiveDataTypes() > * TestHiveStorage.orderByOnHiveTable() > * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() > > * TestHiveViewsSupport.selectStarFromView() > * TestHiveViewsSupport.useHiveAndSelectStarFromView() > * TestHiveViewsSupport.viewWithAllSupportedDataTypes() > * > TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6908) Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile()
Igor Guzenko created DRILL-6908: --- Summary: Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile() Key: DRILL-6908 URL: https://issues.apache.org/jira/browse/DRILL-6908 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.16.0 Reporter: Igor Guzenko For improving Hive tests readability it would be nice to rewrite some cumbersome tests to use CSVTestBuilder for assertions. This issue was created because it requires a lot of changes for different Hive tests and also some additional time may be necessary to ensure that CSVTestBuilder can read all datatypes (dates with different formats, floating point numbers, big decimals etc.) into correct values for assertions. Below is list of test methods to be rewritten: * TestHiveStorage.readAllSupportedHiveDataTypes() * TestHiveStorage.orderByOnHiveTable() * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() * TestHiveViewsSupport.selectStarFromView() * TestHiveViewsSupport.useHiveAndSelectStarFromView() * TestHiveViewsSupport.viewWithAllSupportedDataTypes() * TestInfoSchemaOnHiveStorage.showTablesFromDb() * TestInfoSchemaOnHiveStorage.showDatabases() * TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema() * TestInfoSchemaOnHiveStorage.showInfoSchema() * TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6908) Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile()
[ https://issues.apache.org/jira/browse/DRILL-6908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6908: Issue Type: Task (was: Improvement) > Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile() > -- > > Key: DRILL-6908 > URL: https://issues.apache.org/jira/browse/DRILL-6908 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Igor Guzenko >Priority: Minor > > For improving Hive tests readability it would be nice to rewrite some > cumbersome tests to use CSVTestBuilder for assertions. This issue was created > because it requires a lot of changes for different Hive tests and also some > additional time may be necessary to ensure that CSVTestBuilder can read all > datatypes (dates with different formats, floating point numbers, big decimals > etc.) into correct values for assertions. > Below is list of test methods to be rewritten: > * TestHiveStorage.readAllSupportedHiveDataTypes() > * TestHiveStorage.orderByOnHiveTable() > * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() > > * TestHiveViewsSupport.selectStarFromView() > * TestHiveViewsSupport.useHiveAndSelectStarFromView() > * TestHiveViewsSupport.viewWithAllSupportedDataTypes() > > * TestInfoSchemaOnHiveStorage.showTablesFromDb() > * TestInfoSchemaOnHiveStorage.showDatabases() > * > TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema() > * TestInfoSchemaOnHiveStorage.showInfoSchema() > > * > TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6908) Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile()
[ https://issues.apache.org/jira/browse/DRILL-6908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Guzenko updated DRILL-6908: Description: For improving Hive tests readability it would be nice to rewrite some cumbersome tests to use CSVTestBuilder for assertions. This issue was created because it requires a lot of changes for different Hive tests and also some additional time may be necessary to ensure that CSVTestBuilder can read all datatypes (dates with different formats, floating point numbers, big decimals etc.) into correct values for assertions. Below is list of test methods to be rewritten: * TestHiveStorage.readAllSupportedHiveDataTypes() * TestHiveStorage.orderByOnHiveTable() * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() * TestHiveViewsSupport.selectStarFromView() * TestHiveViewsSupport.useHiveAndSelectStarFromView() * TestHiveViewsSupport.viewWithAllSupportedDataTypes() * TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() was: For improving Hive tests readability it would be nice to rewrite some cumbersome tests to use CSVTestBuilder for assertions. This issue was created because it requires a lot of changes for different Hive tests and also some additional time may be necessary to ensure that CSVTestBuilder can read all datatypes (dates with different formats, floating point numbers, big decimals etc.) into correct values for assertions. Below is list of test methods to be rewritten: * TestHiveStorage.readAllSupportedHiveDataTypes() * TestHiveStorage.orderByOnHiveTable() * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() * TestHiveViewsSupport.selectStarFromView() * TestHiveViewsSupport.useHiveAndSelectStarFromView() * TestHiveViewsSupport.viewWithAllSupportedDataTypes() * TestInfoSchemaOnHiveStorage.showTablesFromDb() * TestInfoSchemaOnHiveStorage.showDatabases() * TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema() * TestInfoSchemaOnHiveStorage.showInfoSchema() * TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() > Hive Testing: Replace cumbersome assertions with TestBuilder.csvBaselineFile() > -- > > Key: DRILL-6908 > URL: https://issues.apache.org/jira/browse/DRILL-6908 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.16.0 >Reporter: Igor Guzenko >Priority: Minor > > For improving Hive tests readability it would be nice to rewrite some > cumbersome tests to use CSVTestBuilder for assertions. This issue was created > because it requires a lot of changes for different Hive tests and also some > additional time may be necessary to ensure that CSVTestBuilder can read all > datatypes (dates with different formats, floating point numbers, big decimals > etc.) into correct values for assertions. > Below is list of test methods to be rewritten: > * TestHiveStorage.readAllSupportedHiveDataTypes() > * TestHiveStorage.orderByOnHiveTable() > * TestHiveStorage.readingFromSmallTableWithSkipHeaderAndFooter() > > * TestHiveViewsSupport.selectStarFromView() > * TestHiveViewsSupport.useHiveAndSelectStarFromView() > * TestHiveViewsSupport.viewWithAllSupportedDataTypes() > > * > TestHiveDrillNativeParquetReader.testReadAllSupportedHiveDataTypesNativeParquet() > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6700) Graceful shutdown from command line against parquet tables fails
[ https://issues.apache.org/jira/browse/DRILL-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-6700: --- Assignee: Venkata Jyothsna Donapati > Graceful shutdown from command line against parquet tables fails > - > > Key: DRILL-6700 > URL: https://issues.apache.org/jira/browse/DRILL-6700 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.14.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Major > Attachments: drillbit.log > > > Trigger a graceful shutdown from the command line while a query against > parquet table is running would fail with the following error: > "Error: SYSTEM ERROR: IOException: Filesystem closed > Fragment 5:11" > This does not occur if graceful shutdown is triggered from the webUI. > Attached is the drillbit.log. > [^drillbit.log] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6894) CTAS and CTTAS are not working on S3 storage when cache is disabled
[
https://issues.apache.org/jira/browse/DRILL-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6894:
Labels: ready-to-commit (was: )
> CTAS and CTTAS are not working on S3 storage when cache is disabled
> ---
>
> Key: DRILL-6894
> URL: https://issues.apache.org/jira/browse/DRILL-6894
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Denys Ordynskiy
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
> Attachments: CTAS_drillbit.log, CTAS_drillbit_queries.json,
> CTAS_sqlline.log, CTTAS_drillbit.log, CTTAS_drillbit_queries.json,
> CTTAS_sqlline.log, s3.json
>
>
> When S3 storage plugin option "fs.s3a.impl.disable.cache" is true in the
> config section and Drill is restarted,
> CTAS and CTTAS queries throwing error in Sqlline:
> *create temporary table s3.tmp.`cttastblwithcache2` as select * from
> cp.`employee.json`;*
> {color:#d04437}Error: SYSTEM ERROR: SdkClientException: Unable to load AWS
> credentials from any provider in the chain
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: 8e386b68-d4fb-4cc6-ba1c-fb41ae0cc9ed on maprhost:31010]
> (state=,code=0){color}
> *create table s3.tmp.`ctastblwithcache` as select * from cp.`employee.json`;*
> {color:#d04437}Error: SYSTEM ERROR: SdkClientException: Unable to load AWS
> credentials from any provider in the chain
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: 4346d300-44be-4f17-90b6-4f3a0db0a148 on maprhost:31010]
> (state=,code=0){color}
> Logs and my storage plugin are in the attachments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6894) CTAS and CTTAS are not working on S3 storage when cache is disabled
[
https://issues.apache.org/jira/browse/DRILL-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6894:
Reviewer: Arina Ielchiieva
> CTAS and CTTAS are not working on S3 storage when cache is disabled
> ---
>
> Key: DRILL-6894
> URL: https://issues.apache.org/jira/browse/DRILL-6894
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Denys Ordynskiy
>Assignee: Bohdan Kazydub
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: CTAS_drillbit.log, CTAS_drillbit_queries.json,
> CTAS_sqlline.log, CTTAS_drillbit.log, CTTAS_drillbit_queries.json,
> CTTAS_sqlline.log, s3.json
>
>
> When S3 storage plugin option "fs.s3a.impl.disable.cache" is true in the
> config section and Drill is restarted,
> CTAS and CTTAS queries throwing error in Sqlline:
> *create temporary table s3.tmp.`cttastblwithcache2` as select * from
> cp.`employee.json`;*
> {color:#d04437}Error: SYSTEM ERROR: SdkClientException: Unable to load AWS
> credentials from any provider in the chain
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: 8e386b68-d4fb-4cc6-ba1c-fb41ae0cc9ed on maprhost:31010]
> (state=,code=0){color}
> *create table s3.tmp.`ctastblwithcache` as select * from cp.`employee.json`;*
> {color:#d04437}Error: SYSTEM ERROR: SdkClientException: Unable to load AWS
> credentials from any provider in the chain
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: 4346d300-44be-4f17-90b6-4f3a0db0a148 on maprhost:31010]
> (state=,code=0){color}
> Logs and my storage plugin are in the attachments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-540) Allow querying hive views in Drill
[
https://issues.apache.org/jira/browse/DRILL-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Guzenko updated DRILL-540:
---
Description:
Currently hive views cannot be queried from drill.
This Jira aims to add support for Hive views in Drill.
*Implementation details:*
# Drill persists it's views metadata in file with suffix .view.drill using
json format. For example:
{noformat}
{
"name" : "view_from_calcite_1_4",
"sql" : "SELECT * FROM `cp`.`store.json`WHERE `store_id` = 0",
"fields" : [ {
"name" : "*",
"type" : "ANY",
"isNullable" : true
} ],
"workspaceSchemaPath" : [ "dfs", "tmp" ]
}
{noformat}
Later Drill parses the metadata and uses it to treat view names in SQL as a
subquery.
2. In Apache Hive metadata about views is stored in similar way to
tables. Below is example from metastore.TBLS :
{noformat}
TBL_ID |CREATE_TIME |DB_ID |LAST_ACCESS_TIME |OWNER |RETENTION |SD_ID |TBL_NAME
|TBL_TYPE |VIEW_EXPANDED_TEXT |
---||--|-|--|--|--|--|--|---|
2 |1542111078 |1 |0|mapr |0 |2 |cview
|VIRTUAL_VIEW |SELECT COUNT(*) FROM `default`.`customers` |
{noformat}
3. So in Hive metastore views are considered as tables of special type.
And main benefit is that we also have expanded SQL definition of views (just
like in view.drill files). Also reading of the metadata is already implemented
in Drill with help of thrift Metastore API.
4. To enable querying of Hive views we'll reuse existing code for Drill
views as much as possible. First in *_HiveSchemaFactory.getDrillTable_* for
_*HiveReadEntry*_ we'll convert the metadata to instance of _*View*_ (_which is
actually model for data persisted in .view.drill files_) and then based on this
instance return new _*DrillViewTable*_. Using this approach drill will handle
hive views the same way as if it was initially defined in Drill and persisted
in .view.drill file.
5. For conversion of Hive types: from _*FieldSchema*_ to _*RelDataType*_
we'll reuse existing code from _*DrillHiveTable*_, so the conversion
functionality will be extracted and used for both (table and view) fields type
conversions.
*Security implications*
Consider simple example case where we have users,
{code:java}
user0 user1 user2
\ /
group12
{code}
and sample db where object names contains user or group who should access them
{code:java}
db_all
tbl_user0
vw_user0
tbl_group12
vw_group12
{code}
There are two Hive authorization modes supported by Drill - SQL Standart and
Strorage Based authorization. For SQL Standart authorization permissions were
granted using SQL:
{code:java}
SET ROLE admin;
GRANT SELECT ON db_all.tbl_user0 TO USER user0;
GRANT SELECT ON db_all.vw_user0 TO USER user0;
CREATE ROLE group12;
GRANT ROLE group12 TO USER user1;
GRANT ROLE group12 TO USER user2;
GRANT SELECT ON db_all.tbl_group12 TO ROLE group12;
GRANT SELECT ON db_all.vw_group12 TO ROLE group12;
{code}
And for Storage based authorization permissions were granted using commands:
{code:java}
hadoop fs -chown user0:user0 /user/hive/warehouse/db_all.db/tbl_user0
hadoop fs -chmod 700 /user/hive/warehouse/db_all.db/tbl_user0
hadoop fs -chmod 750 /user/hive/warehouse/db_all.db/tbl_group12
hadoop fs -chown user1:group12 /user/hive/warehouse/db_all.db/tbl_group12{code}
Then the following table shows us results of queries for both authorization
models.
*SQL
Standart Storage Based Authorization*
||SQL||user0||user1||user2|| ||user0||user1||user2||
|*Queries executed using Drill :*| | | | | | | |
|SHOW TABLES IN hive.db_all;| all| all| all| |Accessibe tables + all
views|Accessibe tables + all views|Accessibe tables + all views|
|SELECT * FROM hive.db_all.tbl_user0;| (/)| (x)| (x)| | (/)|
(x)| (x)|
|SELECT * FROM hive.db_all.vw_user0;| (/)| (x)| (x)| | (/)|
(x)| (x)|
|SELECT * FROM hive.db_all.tbl_group12;| (x)| (/)| (/)| | (x)|
(/)| (/)|
|SELECT * FROM hive.db_all.vw_group12;| (x)| (/)| (/)| | (x)|
(/)| (/)|
|SELECT * FROM INFORMATION_SCHEMA.`TABLES`
WHERE TABLE_SCHEMA = 'hive.db_all';| all| all| all| |Accessibe tables +
all views|Accessibe tables + all views|Accessibe tables + all views|
|DESCRIBE hive.db_all.tbl_user0;| (/)| (x)| (x)| | (/)|
(x)| (x)|
|DESCRIBE hive.db_all.vw_user0;| (/)| (x) | (x)| | (/)|
(/)| (/)|
|DESCRIBE hive.db_all.tbl_group12;| (x)| (/)| (/)| | (x)|
(/) | (/)|
|DESCRIBE hive.db_all.vw_group12;| (x)| (/)| (/)| | (/)|
(/)|
[jira] [Updated] (DRILL-6907) Fix hive-exec-shaded classes recognition in IntelliJ IDEA
[ https://issues.apache.org/jira/browse/DRILL-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6907: Fix Version/s: 1.16.0 > Fix hive-exec-shaded classes recognition in IntelliJ IDEA > - > > Key: DRILL-6907 > URL: https://issues.apache.org/jira/browse/DRILL-6907 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build Test >Affects Versions: 1.14.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Minor > Fix For: 1.16.0 > > > Classes from `hive-exec-shaded` module are not recognized by IntelliJ IDEA. > The solution is to attach the shaded jar to the project using > `build-helper-maven-plugin`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6907) Fix hive-exec-shaded classes recognition in IntelliJ IDEA
[ https://issues.apache.org/jira/browse/DRILL-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6907: Issue Type: Task (was: Bug) > Fix hive-exec-shaded classes recognition in IntelliJ IDEA > - > > Key: DRILL-6907 > URL: https://issues.apache.org/jira/browse/DRILL-6907 > Project: Apache Drill > Issue Type: Task > Components: Tools, Build Test >Affects Versions: 1.14.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Minor > Fix For: 1.16.0 > > > Classes from `hive-exec-shaded` module are not recognized by IntelliJ IDEA. > The solution is to attach the shaded jar to the project using > `build-helper-maven-plugin`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6907) Fix hive-exec-shaded classes recognition in IntelliJ IDEA
[ https://issues.apache.org/jira/browse/DRILL-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6907: Reviewer: Vitalii Diravka > Fix hive-exec-shaded classes recognition in IntelliJ IDEA > - > > Key: DRILL-6907 > URL: https://issues.apache.org/jira/browse/DRILL-6907 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build Test >Affects Versions: 1.14.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Minor > Fix For: 1.16.0 > > > Classes from `hive-exec-shaded` module are not recognized by IntelliJ IDEA. > The solution is to attach the shaded jar to the project using > `build-helper-maven-plugin`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6907) Fix hive-exec-shaded classes recognition in IntelliJ IDEA
Volodymyr Vysotskyi created DRILL-6907: -- Summary: Fix hive-exec-shaded classes recognition in IntelliJ IDEA Key: DRILL-6907 URL: https://issues.apache.org/jira/browse/DRILL-6907 Project: Apache Drill Issue Type: Bug Components: Tools, Build Test Affects Versions: 1.14.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Classes from `hive-exec-shaded` module are not recognized by IntelliJ IDEA. The solution is to attach the shaded jar to the project using `build-helper-maven-plugin`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
