[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392733#comment-17392733
]
lixu commented on FLINK-23096:
--
{code:java}
//代码占位符
java.lang.IllegalArgumentException: Pathname
/C:/Users/merit/AppData/Local/Temp/merit/b6b954c0-78b8-458b-be35-191dcd94d535
from
C:/Users/merit/AppData/Local/Temp/merit/b6b954c0-78b8-458b-be35-191dcd94d535 is
not a valid DFS filename.java.lang.IllegalArgumentException: Pathname
/C:/Users/merit/AppData/Local/Temp/merit/b6b954c0-78b8-458b-be35-191dcd94d535
from
C:/Users/merit/AppData/Local/Temp/merit/b6b954c0-78b8-458b-be35-191dcd94d535 is
not a valid DFS filename. at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:196)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:105)
at
org.apache.hadoop.hdfs.DistributedFileSystem$12.doCall(DistributedFileSystem.java:638)
at
org.apache.hadoop.hdfs.DistributedFileSystem$12.doCall(DistributedFileSystem.java:634)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:634)
at
org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:229)
at
org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:108)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
{code}
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.2
>
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkS
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392631#comment-17392631
]
luoyuxia commented on FLINK-23096:
--
[~leexu] Actual, path.getFileSystem with return HdfsFileSystem or
LocalFileSystem according to what the path is. It'll get LocalFileSystem for
LocalSessionPath, there should be error when delete the LocalSessionPath.
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.2
>
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> // @see org.apache.hadoop.hive.ql.session.SessionState.setCurrentSessionState
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> {code}
>
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391911#comment-17391911
]
lixu commented on FLINK-23096:
--
LocalSessionPath is FileSystem local path;
path.getFileSystem(hiveConf) is HdfsFileSystem,so delete error.
I think , use LocalFileSystem to delete LocalSessionPath.
{code:java}
//代码占位符
org.apache.hadoop.hive.ql.session.SessionState
private static void createPath(HiveConf conf, Path path, String permission,
boolean isLocal, boolean isCleanUp) throws IOException {
FsPermission fsPermission = new FsPermission(permission);
Object fs;
if (isLocal) {
fs = FileSystem.getLocal(conf);
} else {
fs = path.getFileSystem(conf);
}
if (!((FileSystem)fs).exists(path)) {
((FileSystem)fs).mkdirs(path, fsPermission);
String dirType = isLocal ? "local" : "HDFS";
LOG.info("Created " + dirType + " directory: " + path.toString());
}
if (isCleanUp) {
((FileSystem)fs).deleteOnExit(path);
}
}
org.apache.flink.table.planner.delegation.hive.HiveParser
private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
}
{code}
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.2
>
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368590#comment-17368590
]
Rui Li commented on FLINK-23096:
This is actually a different issue from FLINK-16688, un-linking it.
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.2
>
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> // @see org.apache.hadoop.hive.ql.session.SessionState.setCurrentSessionState
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368547#comment-17368547
]
luoyuxia commented on FLINK-23096:
--
Hi [~tinny]
As the exception shows,
java.lang.NullPointerException: Conf non-local session path expected to be
non-null
I think the reason is missing the configuration '_hive.hdfs.session.path'.
And for the last exception you show,
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=service, access=WRITE, inode="/tmp":hdfs:supergroup:drwxr-xr-x
I think the reason is you have no right to access(delete) the file.
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
> Labels: pull-request-available
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> //
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367969#comment-17367969
]
shizhengchao commented on FLINK-23096:
--
The *clearSessionState* method catches IOException, which causes the root
cause to be overwritten。When I caught the Exception, I got the root cause：
{code:java}
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=service, access=WRITE, inode="/tmp":hdfs:supergroup:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:280)
...
at
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3678)
at
org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:597)
at
org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
{code}
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367832#comment-17367832
]
shizhengchao commented on FLINK-23096:
--
[~Leonard Xu] [~lirui]
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> // @see org.apache.hadoop.hive.ql.session.SessionState.setCurrentSessionState
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367327#comment-17367327
]
shizhengchao commented on FLINK-23096:
--
This is the location of the code exception I found:
{code:java}
//代码占位符
// @see org.apache.flink.table.planner.delegation.hive.HiveParser
private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
}
// @see org.apache.hadoop.hive.ql.session.SessionState
public static Path getHDFSSessionPath(Configuration conf) {
SessionState ss = SessionState.get();
if (ss == null) {
String sessionPathString = conf.get(HDFS_SESSION_PATH_KEY);
Preconditions.checkNotNull(sessionPathString,
"Conf non-local session path expected to be non-null");
return new Path(sessionPathString);
}
Preconditions.checkNotNull(ss.hdfsSessionPath,
"Non-local session path expected to be non-null");
return ss.hdfsSessionPath;
}
{code}
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreaming
