Repository: incubator-zeppelin Updated Branches: refs/heads/master b5506f76f -> 79059395b
ZEPPELIN-440 HiveInterpreter with multiple configuration This PR enable HiveInterpreter to configure several connection information. For using specific configurations, you should use some tricks like ``%hive(configuration_prefix)``. without it, you can use default settings starting with ``default``. <img width="673" alt="screen shot 2015-11-21 at 1 51 31 am" src="https://cloud.githubusercontent.com/assets/3612566/11305933/835601e8-8ff2-11e5-9632-6fbbd97010c5.png";> <img width="692" alt="screen shot 2015-11-21 at 1 53 59 am" src="https://cloud.githubusercontent.com/assets/3612566/11306006/dd0882c4-8ff2-11e5-9094-e92dc762a7a6.png";> Author: Jongyoul Lee <[email protected]> Closes #455 from jongyoul/ZEPPELIN-440 and squashes the following commits: af4c64b [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Rebased and fixed links of docs d68bcc5 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed typos on docs fc33a30 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed Apaceh license header - Fixed some codestyle - Removed unused codes and classes 4e01265 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Updated docs for hive b4f1a1c [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed test case to be more general 4867c6f [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed test case to be more general dcb65ae [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Revised executeSql from Postgresql.executeSql f2b61c0 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Restored ParallelScheduler from FIFOScheduler 2616050 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Restored some tests from the beginning 5fa4fd7 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed getScriptBody return after trimming 8aa7601 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed some tests 85e4914 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed style 9293d78 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Fixed linefeed to variable df9f3cb [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Remove commented codes - Fixed license part 87ee87a [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Revert log4j.properties 97bfa65 [Jongyoul Lee] ZEPPELIN-440 HiveInterpreter with multiple configuration - Enable multiple connection properties Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/79059395 Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/79059395 Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/79059395 Branch: refs/heads/master Commit: 79059395bc0c2b539973c6758f61d6abf3403d98 Parents: b5506f7 Author: Jongyoul Lee <[email protected]> Authored: Mon Nov 30 19:15:33 2015 +0900 Committer: Jongyoul Lee <[email protected]> Committed: Fri Dec 4 10:21:37 2015 +0900 ---------------------------------------------------------------------- docs/_includes/themes/zeppelin/_navigation.html | 2 +- docs/docs.md | 2 +- docs/interpreter/hive.md | 105 ++ hive/pom.xml | 6 + .../apache/zeppelin/hive/HiveInterpreter.java | 342 ++-- .../zeppelin/hive/HiveInterpreterTest.java | 1726 ++---------------- .../interpreter/remote/RemoteInterpreter.java | 1 + .../remote/RemoteInterpreterServer.java | 1 + .../org/apache/zeppelin/notebook/Paragraph.java | 6 +- .../apache/zeppelin/notebook/ParagraphTest.java | 38 + 10 files changed, 532 insertions(+), 1697 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/docs/_includes/themes/zeppelin/_navigation.html ---------------------------------------------------------------------- diff --git a/docs/_includes/themes/zeppelin/_navigation.html b/docs/_includes/themes/zeppelin/_navigation.html index 73e856a..440ce42 100644 --- a/docs/_includes/themes/zeppelin/_navigation.html +++ b/docs/_includes/themes/zeppelin/_navigation.html @@ -39,7 +39,7 @@ <li><a href="{{BASE_PATH}}/interpreter/cassandra.html">Cassandra</a></li> <li><a href="{{BASE_PATH}}/interpreter/flink.html">Flink</a></li> <li><a href="{{BASE_PATH}}/interpreter/geode.html">Geode</a></li> - <li><a href="{{BASE_PATH}}/pleasecontribute.html">Hive</a></li> + <li><a href="{{BASE_PATH}}/interpreter/hive.html">Hive</a></li> <li><a href="{{BASE_PATH}}/interpreter/ignite.html">Ignite</a></li> <li><a href="{{BASE_PATH}}/interpreter/lens.html">Lens</a></li> <li><a href="{{BASE_PATH}}/interpreter/markdown.html">Markdown</a></li> http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/docs/docs.md ---------------------------------------------------------------------- diff --git a/docs/docs.md b/docs/docs.md index 0eebfcf..97cba96 100644 --- a/docs/docs.md +++ b/docs/docs.md @@ -36,7 +36,7 @@ limitations under the License. * [cassandra](./interpreter/cassandra.html) * [flink](./interpreter/flink.html) * [geode](./interpreter/geode.html) -* [hive](./pleasecontribute.html) +* [hive](./interpreter/hive.html) * [ignite](./interpreter/ignite.html) * [lens](./interpreter/lens.html) * [md](./interpreter/markdown.html) http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/docs/interpreter/hive.md ---------------------------------------------------------------------- diff --git a/docs/interpreter/hive.md b/docs/interpreter/hive.md new file mode 100644 index 0000000..b37c421 --- /dev/null +++ b/docs/interpreter/hive.md @@ -0,0 +1,105 @@ +--- +layout: page +title: "Hive Interpreter" +description: "" +group: manual +--- +{% include JB/setup %} + + +## Hive Interpreter for Apache Zeppelin + +### Configuration + +<br/> +<table class="table-configuration"> + <tr> + <th>Property</th> + <th>Default</th> + <th>Description</th> + </tr> + <tr> + <td>default.driver</td> + <td>org.apache.hive.jdbc.HiveDriver</td> + <td>Class path of JDBC driver</td> + </tr> + <tr> + <td>default.url</td> + <td>jdbc:hive2://localhost:10000</td> + <td>Url for connection</td> + </tr> + <tr> + <td>default.user</td> + <td></td> + <td><b>(Optional)</b>Username of the connection</td> + </tr> + <tr> + <td>default.password</td> + <td></td> + <td><b>(Optional)</b>Password of the connection</td> + </tr> + <tr> + <td>default.xxx</td> + <td></td> + <td><b>(Optional)</b>Other properties used by the driver</td> + </tr> + <tr> + <td>${prefix}.driver</td> + <td></td> + <td>Driver class path of `%hive(${prefix})`</td> + </tr> + <tr> + <td>${prefix}.url</td> + <td></td> + <td>Url of `%hive(${prefix})`</td> + </tr> + <tr> + <td>${prefix}.user</td> + <td></td> + <td><b>(Optional)</b>Username of the connection of `%hive(${prefix})`</td> + </tr> + <tr> + <td>${prefix}.password</td> + <td></td> + <td><b>(Optional)</b>Password of the connection of `%hive(${prefix})`</td> + </tr> + <tr> + <td>${prefix}.xxx</td> + <td></td> + <td><b>(Optional)</b>Other properties used by the driver of `%hive(${prefix})`</td> + </tr> +</table> + +This interpreter provides multiple configuration with ${prefix}. User can set a multiple connection properties by this prefix. It can be used like `%hive(${prefix})`. + +### How to use + +Basically, you can use + +```sql +%hive +select * from my_table; +``` + +or + +```sql +%hive(etl) +-- 'etl' is a ${prefix} +select * from my_table; +``` + +You can also run multiple queries up to 10 by default. Changing these settings is not implemented yet. + +#### Apply Zeppelin Dynamic Forms + +You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html) inside your queries. You can use both the `text input` and `select form` parameterization features + +```sql +%hive +SELECT ${group_by}, count(*) as count +FROM retail_demo.order_lineitems_pxf +GROUP BY ${group_by=product_id,product_id|product_name|customer_id|store_id} +ORDER BY count ${order=DESC,DESC|ASC} +LIMIT ${limit=10}; +``` http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/hive/pom.xml ---------------------------------------------------------------------- diff --git a/hive/pom.xml b/hive/pom.xml index f4867d3..ca41d38 100644 --- a/hive/pom.xml +++ b/hive/pom.xml @@ -87,6 +87,12 @@ <artifactId>junit</artifactId> <scope>test</scope> </dependency> + <dependency> + <groupId>com.h2database</groupId> + <artifactId>h2</artifactId> + <version>1.4.190</version> + <scope>test</scope> + </dependency> </dependencies> <build> http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/hive/src/main/java/org/apache/zeppelin/hive/HiveInterpreter.java ---------------------------------------------------------------------- diff --git a/hive/src/main/java/org/apache/zeppelin/hive/HiveInterpreter.java b/hive/src/main/java/org/apache/zeppelin/hive/HiveInterpreter.java index 5c3dee3..b1f3339 100644 --- a/hive/src/main/java/org/apache/zeppelin/hive/HiveInterpreter.java +++ b/hive/src/main/java/org/apache/zeppelin/hive/HiveInterpreter.java @@ -1,13 +1,12 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at * - * http://www.apache.org/licenses/LICENSE-2.0 + * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, @@ -15,6 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ + package org.apache.zeppelin.hive; import java.sql.Connection; @@ -23,10 +23,13 @@ import java.sql.ResultSet; import java.sql.ResultSetMetaData; import java.sql.SQLException; import java.sql.Statement; +import java.util.HashMap; +import java.util.HashSet; import java.util.List; +import java.util.Map; import java.util.Properties; +import java.util.Set; -import org.apache.commons.lang.StringUtils; import org.apache.zeppelin.interpreter.Interpreter; import org.apache.zeppelin.interpreter.InterpreterContext; import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder; @@ -37,152 +40,284 @@ import org.apache.zeppelin.scheduler.SchedulerFactory; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.apache.commons.lang.StringUtils.containsIgnoreCase; + /** * Hive interpreter for Zeppelin. */ public class HiveInterpreter extends Interpreter { Logger logger = LoggerFactory.getLogger(HiveInterpreter.class); - int commandTimeOut = 600000; - static final String HIVESERVER_URL = "hive.hiveserver2.url"; - static final String HIVESERVER_USER = "hive.hiveserver2.user"; - static final String HIVESERVER_PASSWORD = "hive.hiveserver2.password"; + static final String COMMON_KEY = "common"; + static final String MAX_LINE_KEY = "max_count"; + static final String MAX_LINE_DEFAULT = "1000"; + + static final String DEFAULT_KEY = "default"; + static final String DRIVER_KEY = "driver"; + static final String URL_KEY = "url"; + static final String USER_KEY = "user"; + static final String PASSWORD_KEY = "password"; + static final String DOT = "."; + + static final char TAB = '\t'; + static final char NEWLINE = '\n'; + static final String EXPLAIN_PREDICATE = "EXPLAIN "; + static final String TABLE_MAGIC_TAG = "%table "; + static final String UPDATE_COUNT_HEADER = "Update Count"; + + static final String COMMON_MAX_LINE = COMMON_KEY + DOT + MAX_LINE_KEY; + + static final String DEFAULT_DRIVER = DEFAULT_KEY + DOT + DRIVER_KEY; + static final String DEFAULT_URL = DEFAULT_KEY + DOT + URL_KEY; + static final String DEFAULT_USER = DEFAULT_KEY + DOT + USER_KEY; + static final String DEFAULT_PASSWORD = DEFAULT_KEY + DOT + PASSWORD_KEY; + + private final HashMap<String, Properties> propertiesMap; + private final Map<String, Connection> keyConnectionMap; + private final Map<String, Statement> paragraphIdStatementMap; static { Interpreter.register( - "hql", - "hive", - HiveInterpreter.class.getName(), - new InterpreterPropertyBuilder() - .add(HIVESERVER_URL, "jdbc:hive2://localhost:10000", "The URL for HiveServer2.") - .add(HIVESERVER_USER, "hive", "The hive user") - .add(HIVESERVER_PASSWORD, "", "The password for the hive user").build()); + "hql", + "hive", + HiveInterpreter.class.getName(), + new InterpreterPropertyBuilder() + .add(COMMON_MAX_LINE, MAX_LINE_DEFAULT, "Maximum line of results") + .add(DEFAULT_DRIVER, "org.apache.hive.jdbc.HiveDriver", "Hive JDBC driver") + .add(DEFAULT_URL, "jdbc:hive2://localhost:10000", "The URL for HiveServer2.") + .add(DEFAULT_USER, "hive", "The hive user") + .add(DEFAULT_PASSWORD, "", "The password for the hive user").build()); } public HiveInterpreter(Properties property) { super(property); + propertiesMap = new HashMap<>(); + keyConnectionMap = new HashMap<>(); + paragraphIdStatementMap = new HashMap<>(); } - Connection jdbcConnection; - Exception exceptionOnConnect; - - //Test only method - public Connection getJdbcConnection() - throws SQLException { - String url = getProperty(HIVESERVER_URL); - String user = getProperty(HIVESERVER_USER); - String password = getProperty(HIVESERVER_PASSWORD); - - return DriverManager.getConnection(url, user, password); + public HashMap<String, Properties> getPropertiesMap() { + return propertiesMap; } @Override public void open() { - logger.info("Jdbc open connection called!"); - try { - String driverName = "org.apache.hive.jdbc.HiveDriver"; - Class.forName(driverName); - } catch (ClassNotFoundException e) { - logger.error("Can not open connection", e); - exceptionOnConnect = e; - return; + logger.debug("property: {}", property); + + for (String propertyKey : property.stringPropertyNames()) { + logger.debug("propertyKey: {}", propertyKey); + String[] keyValue = propertyKey.split("\\.", 2); + if (2 == keyValue.length) { + logger.debug("key: {}, value: {}", keyValue[0], keyValue[1]); + Properties prefixProperties; + if (propertiesMap.containsKey(keyValue[0])) { + prefixProperties = propertiesMap.get(keyValue[0]); + } else { + prefixProperties = new Properties(); + propertiesMap.put(keyValue[0], prefixProperties); + } + prefixProperties.put(keyValue[1], property.getProperty(propertyKey)); + } } - try { - jdbcConnection = getJdbcConnection(); - exceptionOnConnect = null; - logger.info("Successfully created Jdbc connection"); + + Set<String> removeKeySet = new HashSet<>(); + for (String key : propertiesMap.keySet()) { + if (!COMMON_KEY.equals(key)) { + Properties properties = propertiesMap.get(key); + if (!properties.containsKey(DRIVER_KEY) || !properties.containsKey(URL_KEY)) { + logger.error("{} will be ignored. {}.{} and {}.{} is mandatory.", + key, DRIVER_KEY, key, key, URL_KEY); + removeKeySet.add(key); + } + } } - catch (SQLException e) { - logger.error("Cannot open connection", e); - exceptionOnConnect = e; + + for (String key : removeKeySet) { + propertiesMap.remove(key); } + + logger.debug("propertiesMap: {}", propertiesMap); } @Override public void close() { try { - if (jdbcConnection != null) { - jdbcConnection.close(); + for (Statement statement : paragraphIdStatementMap.values()) { + statement.close(); } + paragraphIdStatementMap.clear(); + + for (Connection connection : keyConnectionMap.values()) { + connection.close(); + } + keyConnectionMap.clear(); + } catch (SQLException e) { + logger.error("Error while closing...", e); } - catch (SQLException e) { - logger.error("Cannot close connection", e); + } + + public Connection getConnection(String propertyKey) throws ClassNotFoundException, SQLException { + Connection connection = null; + if (keyConnectionMap.containsKey(propertyKey)) { + connection = keyConnectionMap.get(propertyKey); + if (connection.isClosed() || !connection.isValid(10)) { + connection.close(); + connection = null; + keyConnectionMap.remove(propertyKey); + } } - finally { - jdbcConnection = null; - exceptionOnConnect = null; + if (null == connection) { + Properties properties = propertiesMap.get(propertyKey); + Class.forName(properties.getProperty(DRIVER_KEY)); + String url = properties.getProperty(URL_KEY); + String user = properties.getProperty(USER_KEY); + String password = properties.getProperty(PASSWORD_KEY); + if (null != user && null != password) { + connection = DriverManager.getConnection(url, user, password); + } else { + connection = DriverManager.getConnection(url, properties); + } + keyConnectionMap.put(propertyKey, connection); } + return connection; } - Statement currentStatement; - private InterpreterResult executeSql(String sql) { - try { - if (exceptionOnConnect != null) { - return new InterpreterResult(Code.ERROR, exceptionOnConnect.getMessage()); + public Statement getStatement(String propertyKey, String paragraphId) + throws SQLException, ClassNotFoundException { + Statement statement = null; + if (paragraphIdStatementMap.containsKey(paragraphId)) { + statement = paragraphIdStatementMap.get(paragraphId); + if (statement.isClosed()) { + statement = null; + paragraphIdStatementMap.remove(paragraphId); } - currentStatement = jdbcConnection.createStatement(); - StringBuilder msg = null; - if (StringUtils.containsIgnoreCase(sql, "EXPLAIN ")) { - //return the explain as text, make this visual explain later + } + if (null == statement) { + statement = getConnection(propertyKey).createStatement(); + paragraphIdStatementMap.put(paragraphId, statement); + } + return statement; + } + + public InterpreterResult executeSql(String propertyKey, String sql, + InterpreterContext interpreterContext) { + String paragraphId = interpreterContext.getParagraphId(); + + try { + + Statement statement = getStatement(propertyKey, paragraphId); + + statement.setMaxRows(getMaxResult()); + + StringBuilder msg; + + if (containsIgnoreCase(sql, EXPLAIN_PREDICATE)) { msg = new StringBuilder(); + } else { + msg = new StringBuilder(TABLE_MAGIC_TAG); } - else { - msg = new StringBuilder("%table "); - } - ResultSet res = currentStatement.executeQuery(sql); + + ResultSet resultSet = null; + try { - ResultSetMetaData md = res.getMetaData(); - for (int i = 1; i < md.getColumnCount() + 1; i++) { - if (i == 1) { + boolean isResultSetAvailable = statement.execute(sql); + + if (isResultSetAvailable) { + resultSet = statement.getResultSet(); + + ResultSetMetaData md = resultSet.getMetaData(); + + for (int i = 1; i < md.getColumnCount() + 1; i++) { + if (i > 1) { + msg.append(TAB); + } msg.append(md.getColumnName(i)); - } else { - msg.append("\t" + md.getColumnName(i)); } - } - msg.append("\n"); - while (res.next()) { - for (int i = 1; i < md.getColumnCount() + 1; i++) { - msg.append(res.getString(i) + "\t"); + msg.append(NEWLINE); + + int displayRowCount = 0; + while (resultSet.next() && displayRowCount < getMaxResult()) { + for (int i = 1; i < md.getColumnCount() + 1; i++) { + msg.append(resultSet.getString(i)); + if (i != md.getColumnCount()) { + msg.append(TAB); + } + } + msg.append(NEWLINE); + displayRowCount++; } - msg.append("\n"); + } else { + // Response contains either an update count or there are no results. + int updateCount = statement.getUpdateCount(); + msg.append(UPDATE_COUNT_HEADER).append(NEWLINE); + msg.append(updateCount).append(NEWLINE); } - } - finally { + } finally { try { - res.close(); - currentStatement.close(); - } - finally { - currentStatement = null; + if (resultSet != null) { + resultSet.close(); + } + statement.close(); + } finally { + removeStatement(paragraphId); } } - InterpreterResult rett = new InterpreterResult(Code.SUCCESS, msg.toString()); - return rett; - } - catch (SQLException ex) { - logger.error("Can not run " + sql, ex); + return new InterpreterResult(Code.SUCCESS, msg.toString()); + + } catch (SQLException | ClassNotFoundException ex) { + logger.error("Cannot run " + sql, ex); return new InterpreterResult(Code.ERROR, ex.getMessage()); } } + private void removeStatement(String paragraphId) { + paragraphIdStatementMap.remove(paragraphId); + } + @Override public InterpreterResult interpret(String cmd, InterpreterContext contextInterpreter) { - logger.info("Run SQL command '" + cmd + "'"); - return executeSql(cmd); + String propertyKey = getPropertyKey(cmd); + + if (null != propertyKey) { + cmd = cmd.substring(propertyKey.length() + 2); + } else { + propertyKey = DEFAULT_KEY; + } + + cmd = cmd.trim(); + + logger.info("PropertyKey: {}, SQL command: '{}'", propertyKey, cmd); + + return executeSql(propertyKey, cmd, contextInterpreter); + } + + private int getMaxResult() { + return Integer.valueOf( + propertiesMap.get(COMMON_KEY).getProperty(MAX_LINE_KEY, MAX_LINE_DEFAULT)); + } + + public String getPropertyKey(String cmd) { + int firstLineIndex = cmd.indexOf("\n"); + if (-1 == firstLineIndex) { + firstLineIndex = cmd.length(); + } + int configStartIndex = cmd.indexOf("("); + int configLastIndex = cmd.indexOf(")"); + if (configStartIndex != -1 && configLastIndex != -1 + && configLastIndex < firstLineIndex && configLastIndex < firstLineIndex) { + return cmd.substring(configStartIndex + 1, configLastIndex); + } + return null; } @Override public void cancel(InterpreterContext context) { - if (currentStatement != null) { - try { - currentStatement.cancel(); - } - catch (SQLException ex) { - } - finally { - currentStatement = null; - } + String paragraphId = context.getParagraphId(); + try { + paragraphIdStatementMap.get(paragraphId).cancel(); + } catch (SQLException e) { + logger.error("Error while cancelling...", e); } } @@ -198,13 +333,12 @@ public class HiveInterpreter extends Interpreter { @Override public Scheduler getScheduler() { - return SchedulerFactory.singleton().createOrGetFIFOScheduler( - HiveInterpreter.class.getName() + this.hashCode()); + return SchedulerFactory.singleton().createOrGetParallelScheduler( + HiveInterpreter.class.getName() + this.hashCode(), 10); } @Override public List<String> completion(String buf, int cursor) { return null; } - } http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/hive/src/test/java/org/apache/zeppelin/hive/HiveInterpreterTest.java ---------------------------------------------------------------------- diff --git a/hive/src/test/java/org/apache/zeppelin/hive/HiveInterpreterTest.java b/hive/src/test/java/org/apache/zeppelin/hive/HiveInterpreterTest.java index 41ab108..c22080d 100644 --- a/hive/src/test/java/org/apache/zeppelin/hive/HiveInterpreterTest.java +++ b/hive/src/test/java/org/apache/zeppelin/hive/HiveInterpreterTest.java @@ -17,30 +17,50 @@ */ package org.apache.zeppelin.hive; -import static org.junit.Assert.assertEquals; - -import java.io.InputStream; -import java.io.Reader; -import java.math.BigDecimal; -import java.net.URL; -import java.sql.*; -import java.util.Calendar; -import java.util.Map; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Statement; import java.util.Properties; -import java.util.concurrent.Executor; import org.apache.zeppelin.interpreter.InterpreterContext; import org.apache.zeppelin.interpreter.InterpreterResult; import org.junit.After; -import org.junit.Before; +import org.junit.BeforeClass; import org.junit.Test; +import static org.junit.Assert.*; +import static java.lang.String.format; + /** * Hive interpreter unit tests */ public class HiveInterpreterTest { - @Before - public void setUp() throws Exception { + static String jdbcConnection; + + private static String getJdbcConnection() throws IOException { + if(null == jdbcConnection) { + Path tmpDir = Files.createTempDirectory("h2-test-"); + tmpDir.toFile().deleteOnExit(); + jdbcConnection = format("jdbc:h2:%s", tmpDir); + } + return jdbcConnection; + } + @BeforeClass + public static void setUp() throws Exception { + + Class.forName("org.h2.Driver"); + Connection connection = DriverManager.getConnection(getJdbcConnection()); + Statement statement = connection.createStatement(); + statement.execute( + "DROP TABLE IF EXISTS test_table; " + + "CREATE TABLE test_table(id varchar(255), name varchar(255));"); + statement.execute( + "insert into test_table(id, name) values ('a', 'a_name'),('b', 'b_name');" + ); } @After @@ -48,1599 +68,129 @@ public class HiveInterpreterTest { } @Test - public void test() { - HiveInterpreter t = new MockHiveInterpreter(new Properties()); + public void readTest() throws IOException { + Properties properties = new Properties(); + properties.setProperty("common.max_count", "1000"); + properties.setProperty("common.max_retry", "3"); + properties.setProperty("default.driver", "org.h2.Driver"); + properties.setProperty("default.url", getJdbcConnection()); + properties.setProperty("default.user", ""); + properties.setProperty("default.password", ""); + HiveInterpreter t = new HiveInterpreter(properties); t.open(); - //simple select test - InterpreterResult result = t.interpret("select * from t", null); - assertEquals(result.type(), InterpreterResult.Type.TABLE); - - //explain test - result = t.interpret("explain select * from t", null); - assertEquals(result.type(), InterpreterResult.Type.TEXT); - t.close(); - } -} - -class MockHiveInterpreter extends HiveInterpreter { - - public MockHiveInterpreter(Properties property) { - super(property); - } - - @Override - public Connection getJdbcConnection() - throws SQLException { - return new MockConnection(); - } -} - -class MockResultSetMetadata implements ResultSetMetaData { - - @Override - public int getColumnCount() throws SQLException { - return 0; - } - - @Override - public boolean isAutoIncrement(int column) throws SQLException { - return false; - } - - @Override - public boolean isCaseSensitive(int column) throws SQLException { - return false; - } - - @Override - public boolean isSearchable(int column) throws SQLException { - return false; - } - - @Override - public boolean isCurrency(int column) throws SQLException { - return false; - } - - @Override - public int isNullable(int column) throws SQLException { - return 0; - } - - @Override - public boolean isSigned(int column) throws SQLException { - return false; - } - - @Override - public int getColumnDisplaySize(int column) throws SQLException { - return 0; - } - - @Override - public String getColumnLabel(int column) throws SQLException { - return null; - } - - @Override - public String getColumnName(int column) throws SQLException { - return null; - } - - @Override - public String getSchemaName(int column) throws SQLException { - return null; - } - - @Override - public int getPrecision(int column) throws SQLException { - return 0; - } - - @Override - public int getScale(int column) throws SQLException { - return 0; - } - - @Override - public String getTableName(int column) throws SQLException { - return null; - } - - @Override - public String getCatalogName(int column) throws SQLException { - return null; - } - - @Override - public int getColumnType(int column) throws SQLException { - return 0; - } - - @Override - public String getColumnTypeName(int column) throws SQLException { - return null; - } - - @Override - public boolean isReadOnly(int column) throws SQLException { - return false; - } - - @Override - public boolean isWritable(int column) throws SQLException { - return false; - } - - @Override - public boolean isDefinitelyWritable(int column) throws SQLException { - return false; - } - - @Override - public String getColumnClassName(int column) throws SQLException { - return null; - } - - @Override - public <T> T unwrap(Class<T> iface) throws SQLException { - return null; - } - - @Override - public boolean isWrapperFor(Class<?> iface) throws SQLException { - return false; - } -} -class MockResultSet implements ResultSet { - - @Override - public boolean next() throws SQLException { - return false; - } - - @Override - public void close() throws SQLException { - - } - - @Override - public boolean wasNull() throws SQLException { - return false; - } - - @Override - public String getString(int columnIndex) throws SQLException { - return null; - } - - @Override - public boolean getBoolean(int columnIndex) throws SQLException { - return false; - } - - @Override - public byte getByte(int columnIndex) throws SQLException { - return 0; - } - - @Override - public short getShort(int columnIndex) throws SQLException { - return 0; - } - - @Override - public int getInt(int columnIndex) throws SQLException { - return 0; - } - - @Override - public long getLong(int columnIndex) throws SQLException { - return 0; - } - - @Override - public float getFloat(int columnIndex) throws SQLException { - return 0; - } - - @Override - public double getDouble(int columnIndex) throws SQLException { - return 0; - } - - @Override - public BigDecimal getBigDecimal(int columnIndex, int scale) throws SQLException { - return null; - } - - @Override - public byte[] getBytes(int columnIndex) throws SQLException { - return new byte[0]; - } - - @Override - public Date getDate(int columnIndex) throws SQLException { - return null; - } - - @Override - public Time getTime(int columnIndex) throws SQLException { - return null; - } - - @Override - public Timestamp getTimestamp(int columnIndex) throws SQLException { - return null; - } - - @Override - public InputStream getAsciiStream(int columnIndex) throws SQLException { - return null; - } - - @Override - public InputStream getUnicodeStream(int columnIndex) throws SQLException { - return null; - } - - @Override - public InputStream getBinaryStream(int columnIndex) throws SQLException { - return null; - } - - @Override - public String getString(String columnLabel) throws SQLException { - return null; - } - - @Override - public boolean getBoolean(String columnLabel) throws SQLException { - return false; - } - - @Override - public byte getByte(String columnLabel) throws SQLException { - return 0; - } - - @Override - public short getShort(String columnLabel) throws SQLException { - return 0; - } - - @Override - public int getInt(String columnLabel) throws SQLException { - return 0; - } - - @Override - public long getLong(String columnLabel) throws SQLException { - return 0; - } - - @Override - public float getFloat(String columnLabel) throws SQLException { - return 0; - } - - @Override - public double getDouble(String columnLabel) throws SQLException { - return 0; - } - - @Override - public BigDecimal getBigDecimal(String columnLabel, int scale) throws SQLException { - return null; - } - - @Override - public byte[] getBytes(String columnLabel) throws SQLException { - return new byte[0]; - } - - @Override - public Date getDate(String columnLabel) throws SQLException { - return null; - } - - @Override - public Time getTime(String columnLabel) throws SQLException { - return null; - } - - @Override - public Timestamp getTimestamp(String columnLabel) throws SQLException { - return null; - } - - @Override - public InputStream getAsciiStream(String columnLabel) throws SQLException { - return null; - } - - @Override - public InputStream getUnicodeStream(String columnLabel) throws SQLException { - return null; - } - - @Override - public InputStream getBinaryStream(String columnLabel) throws SQLException { - return null; - } - - @Override - public SQLWarning getWarnings() throws SQLException { - return null; - } - - @Override - public void clearWarnings() throws SQLException { - - } - - @Override - public String getCursorName() throws SQLException { - return null; - } - - @Override - public ResultSetMetaData getMetaData() throws SQLException { - return new MockResultSetMetadata(); - } - - @Override - public Object getObject(int columnIndex) throws SQLException { - return null; - } - - @Override - public Object getObject(String columnLabel) throws SQLException { - return null; - } - - @Override - public int findColumn(String columnLabel) throws SQLException { - return 0; - } - - @Override - public Reader getCharacterStream(int columnIndex) throws SQLException { - return null; - } - - @Override - public Reader getCharacterStream(String columnLabel) throws SQLException { - return null; - } - - @Override - public BigDecimal getBigDecimal(int columnIndex) throws SQLException { - return null; - } - - @Override - public BigDecimal getBigDecimal(String columnLabel) throws SQLException { - return null; - } - - @Override - public boolean isBeforeFirst() throws SQLException { - return false; - } - - @Override - public boolean isAfterLast() throws SQLException { - return false; - } - - @Override - public boolean isFirst() throws SQLException { - return false; - } - - @Override - public boolean isLast() throws SQLException { - return false; - } - - @Override - public void beforeFirst() throws SQLException { - - } - - @Override - public void afterLast() throws SQLException { - - } - - @Override - public boolean first() throws SQLException { - return false; - } - - @Override - public boolean last() throws SQLException { - return false; - } - - @Override - public int getRow() throws SQLException { - return 0; - } - - @Override - public boolean absolute(int row) throws SQLException { - return false; - } - - @Override - public boolean relative(int rows) throws SQLException { - return false; - } - - @Override - public boolean previous() throws SQLException { - return false; - } - - @Override - public void setFetchDirection(int direction) throws SQLException { - - } - - @Override - public int getFetchDirection() throws SQLException { - return 0; - } - - @Override - public void setFetchSize(int rows) throws SQLException { - - } - - @Override - public int getFetchSize() throws SQLException { - return 0; - } - - @Override - public int getType() throws SQLException { - return 0; + assertTrue(t.interpret("show databases", new InterpreterContext("", "1", "","", null,null,null,null)).message().contains("SCHEMA_NAME")); + assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n", + t.interpret("select * from test_table", new InterpreterContext("", "1", "","", null,null,null,null)).message()); } - @Override - public int getConcurrency() throws SQLException { - return 0; - } - - @Override - public boolean rowUpdated() throws SQLException { - return false; - } - - @Override - public boolean rowInserted() throws SQLException { - return false; - } - - @Override - public boolean rowDeleted() throws SQLException { - return false; - } - - @Override - public void updateNull(int columnIndex) throws SQLException { - - } - - @Override - public void updateBoolean(int columnIndex, boolean x) throws SQLException { - - } - - @Override - public void updateByte(int columnIndex, byte x) throws SQLException { - - } - - @Override - public void updateShort(int columnIndex, short x) throws SQLException { - - } - - @Override - public void updateInt(int columnIndex, int x) throws SQLException { - - } - - @Override - public void updateLong(int columnIndex, long x) throws SQLException { - - } - - @Override - public void updateFloat(int columnIndex, float x) throws SQLException { - - } - - @Override - public void updateDouble(int columnIndex, double x) throws SQLException { - - } - - @Override - public void updateBigDecimal(int columnIndex, BigDecimal x) throws SQLException { - - } - - @Override - public void updateString(int columnIndex, String x) throws SQLException { - - } - - @Override - public void updateBytes(int columnIndex, byte[] x) throws SQLException { - - } - - @Override - public void updateDate(int columnIndex, Date x) throws SQLException { - - } - - @Override - public void updateTime(int columnIndex, Time x) throws SQLException { - - } - - @Override - public void updateTimestamp(int columnIndex, Timestamp x) throws SQLException { + @Test + public void readTestWithConfiguration() throws IOException { + Properties properties = new Properties(); + properties.setProperty("common.max_count", "1000"); + properties.setProperty("common.max_retry", "3"); + properties.setProperty("default.driver", "wrong.Driver"); + properties.setProperty("default.url", getJdbcConnection()); + properties.setProperty("default.user", ""); + properties.setProperty("default.password", ""); + properties.setProperty("h2.driver", "org.h2.Driver"); + properties.setProperty("h2.url", getJdbcConnection()); + properties.setProperty("h2.user", ""); + properties.setProperty("h2.password", ""); + HiveInterpreter t = new HiveInterpreter(properties); + t.open(); + assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n", + t.interpret("(h2)\n select * from test_table", new InterpreterContext("", "1", "","", null,null,null,null)).message()); } - @Override - public void updateAsciiStream(int columnIndex, InputStream x, int length) throws SQLException { + @Test + public void jdbcRestart() throws IOException, SQLException, ClassNotFoundException { + Properties properties = new Properties(); + properties.setProperty("common.max_count", "1000"); + properties.setProperty("common.max_retry", "3"); + properties.setProperty("default.driver", "org.h2.Driver"); + properties.setProperty("default.url", getJdbcConnection()); + properties.setProperty("default.user", ""); + properties.setProperty("default.password", ""); + HiveInterpreter t = new HiveInterpreter(properties); + t.open(); - } + InterpreterResult interpreterResult = + t.interpret("select * from test_table", new InterpreterContext("", "1", "","", null,null,null,null)); + assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n", interpreterResult.message()); - @Override - public void updateBinaryStream(int columnIndex, InputStream x, int length) throws SQLException { + t.getConnection("default").close(); + interpreterResult = + t.interpret("select * from test_table", new InterpreterContext("", "1", "","", null,null,null,null)); + assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n", interpreterResult.message()); } - @Override - public void updateCharacterStream(int columnIndex, Reader x, int length) throws SQLException { + @Test + public void test() throws IOException { + Properties properties = new Properties(); + properties.setProperty("common.max_count", "1000"); + properties.setProperty("common.max_retry", "3"); + properties.setProperty("default.driver", "org.h2.Driver"); + properties.setProperty("default.url", getJdbcConnection()); + properties.setProperty("default.user", ""); + properties.setProperty("default.password", ""); + HiveInterpreter t = new HiveInterpreter(properties); + t.open(); - } + InterpreterContext interpreterContext = new InterpreterContext(null, "a", null, null, null, null, null, null); - @Override - public void updateObject(int columnIndex, Object x, int scaleOrLength) throws SQLException { + //simple select test + InterpreterResult result = t.interpret("select * from test_table", interpreterContext); + assertEquals(result.type(), InterpreterResult.Type.TABLE); + //explain test + result = t.interpret("explain select * from test_table", interpreterContext); + assertEquals(result.type(), InterpreterResult.Type.TEXT); + t.close(); } - @Override - public void updateObject(int columnIndex, Object x) throws SQLException { - + @Test + public void parseMultiplePropertiesMap() { + Properties properties = new Properties(); + properties.setProperty("common.max_count", "1000"); + properties.setProperty("common.max_retry", "3"); + properties.setProperty("default.driver", "defaultDriver"); + properties.setProperty("default.url", "defaultUri"); + properties.setProperty("default.user", "defaultUser"); + HiveInterpreter hi = new HiveInterpreter(properties); + hi.open(); + assertNotNull("propertiesMap is not null", hi.getPropertiesMap()); + assertNotNull("propertiesMap.get(default) is not null", hi.getPropertiesMap().get("default")); + assertTrue("default exists", "defaultDriver".equals(hi.getPropertiesMap().get("default").getProperty("driver"))); + hi.close(); } - @Override - public void updateNull(String columnLabel) throws SQLException { - + @Test + public void ignoreInvalidSettings() { + Properties properties = new Properties(); + properties.setProperty("common.max_count", "1000"); + properties.setProperty("common.max_retry", "3"); + properties.setProperty("default.driver", "defaultDriver"); + properties.setProperty("default.url", "defaultUri"); + properties.setProperty("default.user", "defaultUser"); + properties.setProperty("presto.driver", "com.facebook.presto.jdbc.PrestoDriver"); + HiveInterpreter hi = new HiveInterpreter(properties); + hi.open(); + assertTrue("default exists", hi.getPropertiesMap().containsKey("default")); + assertFalse("presto doesn't exists", hi.getPropertiesMap().containsKey("presto")); + hi.close(); } - @Override - public void updateBoolean(String columnLabel, boolean x) throws SQLException { - - } - - @Override - public void updateByte(String columnLabel, byte x) throws SQLException { - - } - - @Override - public void updateShort(String columnLabel, short x) throws SQLException { - - } - - @Override - public void updateInt(String columnLabel, int x) throws SQLException { - - } - - @Override - public void updateLong(String columnLabel, long x) throws SQLException { - - } - - @Override - public void updateFloat(String columnLabel, float x) throws SQLException { - - } - - @Override - public void updateDouble(String columnLabel, double x) throws SQLException { - - } - - @Override - public void updateBigDecimal(String columnLabel, BigDecimal x) throws SQLException { - - } - - @Override - public void updateString(String columnLabel, String x) throws SQLException { - - } - - @Override - public void updateBytes(String columnLabel, byte[] x) throws SQLException { - - } - - @Override - public void updateDate(String columnLabel, Date x) throws SQLException { - - } - - @Override - public void updateTime(String columnLabel, Time x) throws SQLException { - - } - - @Override - public void updateTimestamp(String columnLabel, Timestamp x) throws SQLException { - - } - - @Override - public void updateAsciiStream(String columnLabel, InputStream x, int length) throws SQLException { - - } - - @Override - public void updateBinaryStream(String columnLabel, InputStream x, int length) throws SQLException { - - } - - @Override - public void updateCharacterStream(String columnLabel, Reader reader, int length) throws SQLException { - - } - - @Override - public void updateObject(String columnLabel, Object x, int scaleOrLength) throws SQLException { - - } - - @Override - public void updateObject(String columnLabel, Object x) throws SQLException { - - } - - @Override - public void insertRow() throws SQLException { - - } - - @Override - public void updateRow() throws SQLException { - - } - - @Override - public void deleteRow() throws SQLException { - - } - - @Override - public void refreshRow() throws SQLException { - - } - - @Override - public void cancelRowUpdates() throws SQLException { - - } - - @Override - public void moveToInsertRow() throws SQLException { - - } - - @Override - public void moveToCurrentRow() throws SQLException { - - } - - @Override - public Statement getStatement() throws SQLException { - return new MockStatement(); - } - - @Override - public Object getObject(int columnIndex, Map<String, Class<?>> map) throws SQLException { - return null; - } - - @Override - public Ref getRef(int columnIndex) throws SQLException { - return null; - } - - @Override - public Blob getBlob(int columnIndex) throws SQLException { - return null; - } - - @Override - public Clob getClob(int columnIndex) throws SQLException { - return null; - } - - @Override - public Array getArray(int columnIndex) throws SQLException { - return null; - } - - @Override - public Object getObject(String columnLabel, Map<String, Class<?>> map) throws SQLException { - return null; - } - - @Override - public Ref getRef(String columnLabel) throws SQLException { - return null; - } - - @Override - public Blob getBlob(String columnLabel) throws SQLException { - return null; - } - - @Override - public Clob getClob(String columnLabel) throws SQLException { - return null; - } - - @Override - public Array getArray(String columnLabel) throws SQLException { - return null; - } - - @Override - public Date getDate(int columnIndex, Calendar cal) throws SQLException { - return null; - } - - @Override - public Date getDate(String columnLabel, Calendar cal) throws SQLException { - return null; - } - - @Override - public Time getTime(int columnIndex, Calendar cal) throws SQLException { - return null; - } - - @Override - public Time getTime(String columnLabel, Calendar cal) throws SQLException { - return null; - } - - @Override - public Timestamp getTimestamp(int columnIndex, Calendar cal) throws SQLException { - return null; - } - - @Override - public Timestamp getTimestamp(String columnLabel, Calendar cal) throws SQLException { - return null; - } - - @Override - public URL getURL(int columnIndex) throws SQLException { - return null; - } - - @Override - public URL getURL(String columnLabel) throws SQLException { - return null; - } - - @Override - public void updateRef(int columnIndex, Ref x) throws SQLException { - - } - - @Override - public void updateRef(String columnLabel, Ref x) throws SQLException { - - } - - @Override - public void updateBlob(int columnIndex, Blob x) throws SQLException { - - } - - @Override - public void updateBlob(String columnLabel, Blob x) throws SQLException { - - } - - @Override - public void updateClob(int columnIndex, Clob x) throws SQLException { - - } - - @Override - public void updateClob(String columnLabel, Clob x) throws SQLException { - - } - - @Override - public void updateArray(int columnIndex, Array x) throws SQLException { - - } - - @Override - public void updateArray(String columnLabel, Array x) throws SQLException { - - } - - @Override - public RowId getRowId(int columnIndex) throws SQLException { - return null; - } - - @Override - public RowId getRowId(String columnLabel) throws SQLException { - return null; - } - - @Override - public void updateRowId(int columnIndex, RowId x) throws SQLException { - - } - - @Override - public void updateRowId(String columnLabel, RowId x) throws SQLException { - - } - - @Override - public int getHoldability() throws SQLException { - return 0; - } - - @Override - public boolean isClosed() throws SQLException { - return false; - } - - @Override - public void updateNString(int columnIndex, String nString) throws SQLException { - - } - - @Override - public void updateNString(String columnLabel, String nString) throws SQLException { - - } - - @Override - public void updateNClob(int columnIndex, NClob nClob) throws SQLException { - - } - - @Override - public void updateNClob(String columnLabel, NClob nClob) throws SQLException { - - } - - @Override - public NClob getNClob(int columnIndex) throws SQLException { - return null; - } - - @Override - public NClob getNClob(String columnLabel) throws SQLException { - return null; - } - - @Override - public SQLXML getSQLXML(int columnIndex) throws SQLException { - return null; - } - - @Override - public SQLXML getSQLXML(String columnLabel) throws SQLException { - return null; - } - - @Override - public void updateSQLXML(int columnIndex, SQLXML xmlObject) throws SQLException { - - } - - @Override - public void updateSQLXML(String columnLabel, SQLXML xmlObject) throws SQLException { - - } - - @Override - public String getNString(int columnIndex) throws SQLException { - return null; - } - - @Override - public String getNString(String columnLabel) throws SQLException { - return null; - } - - @Override - public Reader getNCharacterStream(int columnIndex) throws SQLException { - return null; - } - - @Override - public Reader getNCharacterStream(String columnLabel) throws SQLException { - return null; - } - - @Override - public void updateNCharacterStream(int columnIndex, Reader x, long length) throws SQLException { - - } - - @Override - public void updateNCharacterStream(String columnLabel, Reader reader, long length) throws SQLException { - - } - - @Override - public void updateAsciiStream(int columnIndex, InputStream x, long length) throws SQLException { - - } - - @Override - public void updateBinaryStream(int columnIndex, InputStream x, long length) throws SQLException { - - } - - @Override - public void updateCharacterStream(int columnIndex, Reader x, long length) throws SQLException { - - } - - @Override - public void updateAsciiStream(String columnLabel, InputStream x, long length) throws SQLException { - - } - - @Override - public void updateBinaryStream(String columnLabel, InputStream x, long length) throws SQLException { - - } - - @Override - public void updateCharacterStream(String columnLabel, Reader reader, long length) throws SQLException { - - } - - @Override - public void updateBlob(int columnIndex, InputStream inputStream, long length) throws SQLException { - - } - - @Override - public void updateBlob(String columnLabel, InputStream inputStream, long length) throws SQLException { - - } - - @Override - public void updateClob(int columnIndex, Reader reader, long length) throws SQLException { - - } - - @Override - public void updateClob(String columnLabel, Reader reader, long length) throws SQLException { - - } - - @Override - public void updateNClob(int columnIndex, Reader reader, long length) throws SQLException { - - } - - @Override - public void updateNClob(String columnLabel, Reader reader, long length) throws SQLException { - - } - - @Override - public void updateNCharacterStream(int columnIndex, Reader x) throws SQLException { - - } - - @Override - public void updateNCharacterStream(String columnLabel, Reader reader) throws SQLException { - - } - - @Override - public void updateAsciiStream(int columnIndex, InputStream x) throws SQLException { - - } - - @Override - public void updateBinaryStream(int columnIndex, InputStream x) throws SQLException { - - } - - @Override - public void updateCharacterStream(int columnIndex, Reader x) throws SQLException { - - } - - @Override - public void updateAsciiStream(String columnLabel, InputStream x) throws SQLException { - - } - - @Override - public void updateBinaryStream(String columnLabel, InputStream x) throws SQLException { - - } - - @Override - public void updateCharacterStream(String columnLabel, Reader reader) throws SQLException { - - } - - @Override - public void updateBlob(int columnIndex, InputStream inputStream) throws SQLException { - - } - - @Override - public void updateBlob(String columnLabel, InputStream inputStream) throws SQLException { - - } - - @Override - public void updateClob(int columnIndex, Reader reader) throws SQLException { - - } - - @Override - public void updateClob(String columnLabel, Reader reader) throws SQLException { - - } - - @Override - public void updateNClob(int columnIndex, Reader reader) throws SQLException { - - } - - @Override - public void updateNClob(String columnLabel, Reader reader) throws SQLException { - - } - - @Override - public <T> T getObject(int columnIndex, Class<T> type) throws SQLException { - return null; - } - - @Override - public <T> T getObject(String columnLabel, Class<T> type) throws SQLException { - return null; - } - - @Override - public <T> T unwrap(Class<T> iface) throws SQLException { - return null; - } - - @Override - public boolean isWrapperFor(Class<?> iface) throws SQLException { - return false; - } -} -class MockStatement implements Statement { - - @Override - public ResultSet executeQuery(String sql) throws SQLException { - return new MockResultSet(); - } - - @Override - public int executeUpdate(String sql) throws SQLException { - return 0; - } - - @Override - public void close() throws SQLException { - - } - - @Override - public int getMaxFieldSize() throws SQLException { - return 0; - } - - @Override - public void setMaxFieldSize(int max) throws SQLException { - - } - - @Override - public int getMaxRows() throws SQLException { - return 0; - } - - @Override - public void setMaxRows(int max) throws SQLException { - - } - - @Override - public void setEscapeProcessing(boolean enable) throws SQLException { - - } - - @Override - public int getQueryTimeout() throws SQLException { - return 0; - } - - @Override - public void setQueryTimeout(int seconds) throws SQLException { - - } - - @Override - public void cancel() throws SQLException { - - } - - @Override - public SQLWarning getWarnings() throws SQLException { - return null; - } - - @Override - public void clearWarnings() throws SQLException { - - } - - @Override - public void setCursorName(String name) throws SQLException { - - } - - @Override - public boolean execute(String sql) throws SQLException { - return false; - } - - @Override - public ResultSet getResultSet() throws SQLException { - return new MockResultSet(); - } - - @Override - public int getUpdateCount() throws SQLException { - return 0; - } - - @Override - public boolean getMoreResults() throws SQLException { - return false; - } - - @Override - public void setFetchDirection(int direction) throws SQLException { - - } - - @Override - public int getFetchDirection() throws SQLException { - return 0; - } - - @Override - public void setFetchSize(int rows) throws SQLException { - - } - - @Override - public int getFetchSize() throws SQLException { - return 0; - } - - @Override - public int getResultSetConcurrency() throws SQLException { - return 0; - } - - @Override - public int getResultSetType() throws SQLException { - return 0; - } - - @Override - public void addBatch(String sql) throws SQLException { - - } - - @Override - public void clearBatch() throws SQLException { - - } - - @Override - public int[] executeBatch() throws SQLException { - return new int[0]; - } - - @Override - public Connection getConnection() throws SQLException { - return null; - } - - @Override - public boolean getMoreResults(int current) throws SQLException { - return false; - } - - @Override - public ResultSet getGeneratedKeys() throws SQLException { - return null; - } - - @Override - public int executeUpdate(String sql, int autoGeneratedKeys) throws SQLException { - return 0; - } - - @Override - public int executeUpdate(String sql, int[] columnIndexes) throws SQLException { - return 0; - } - - @Override - public int executeUpdate(String sql, String[] columnNames) throws SQLException { - return 0; - } - - @Override - public boolean execute(String sql, int autoGeneratedKeys) throws SQLException { - return false; - } - - @Override - public boolean execute(String sql, int[] columnIndexes) throws SQLException { - return false; - } - - @Override - public boolean execute(String sql, String[] columnNames) throws SQLException { - return false; - } - - @Override - public int getResultSetHoldability() throws SQLException { - return 0; - } - - @Override - public boolean isClosed() throws SQLException { - return false; - } - - @Override - public void setPoolable(boolean poolable) throws SQLException { - - } - - @Override - public boolean isPoolable() throws SQLException { - return false; - } - - @Override - public void closeOnCompletion() throws SQLException { - - } - - @Override - public boolean isCloseOnCompletion() throws SQLException { - return false; - } - - @Override - public <T> T unwrap(Class<T> iface) throws SQLException { - return null; - } - - @Override - public boolean isWrapperFor(Class<?> iface) throws SQLException { - return false; - } -} -class MockConnection implements Connection { - - @Override - public Statement createStatement() throws SQLException { - return new MockStatement(); - } - - @Override - public PreparedStatement prepareStatement(String sql) throws SQLException { - return null; - } - - @Override - public CallableStatement prepareCall(String sql) throws SQLException { - return null; - } - - @Override - public String nativeSQL(String sql) throws SQLException { - return null; - } - - @Override - public void setAutoCommit(boolean autoCommit) throws SQLException { - - } - - @Override - public boolean getAutoCommit() throws SQLException { - return false; - } - - @Override - public void commit() throws SQLException { - - } - - @Override - public void rollback() throws SQLException { - - } - - @Override - public void close() throws SQLException { - - } - - @Override - public boolean isClosed() throws SQLException { - return false; - } - - @Override - public DatabaseMetaData getMetaData() throws SQLException { - return null; - } - - @Override - public void setReadOnly(boolean readOnly) throws SQLException { - - } - - @Override - public boolean isReadOnly() throws SQLException { - return false; - } - - @Override - public void setCatalog(String catalog) throws SQLException { - - } - - @Override - public String getCatalog() throws SQLException { - return null; - } - - @Override - public void setTransactionIsolation(int level) throws SQLException { - - } - - @Override - public int getTransactionIsolation() throws SQLException { - return 0; - } - - @Override - public SQLWarning getWarnings() throws SQLException { - return null; - } - - @Override - public void clearWarnings() throws SQLException { - - } - - @Override - public Statement createStatement(int resultSetType, int resultSetConcurrency) throws SQLException { - return null; - } - - @Override - public PreparedStatement prepareStatement(String sql, int resultSetType, int resultSetConcurrency) throws SQLException { - return null; - } - - @Override - public CallableStatement prepareCall(String sql, int resultSetType, int resultSetConcurrency) throws SQLException { - return null; - } - - @Override - public Map<String, Class<?>> getTypeMap() throws SQLException { - return null; - } - - @Override - public void setTypeMap(Map<String, Class<?>> map) throws SQLException { - - } - - @Override - public void setHoldability(int holdability) throws SQLException { - - } - - @Override - public int getHoldability() throws SQLException { - return 0; - } - - @Override - public Savepoint setSavepoint() throws SQLException { - return null; - } - - @Override - public Savepoint setSavepoint(String name) throws SQLException { - return null; - } - - @Override - public void rollback(Savepoint savepoint) throws SQLException { - - } - - @Override - public void releaseSavepoint(Savepoint savepoint) throws SQLException { - - } - - @Override - public Statement createStatement(int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException { - return null; - } - - @Override - public PreparedStatement prepareStatement(String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException { - return null; - } - - @Override - public CallableStatement prepareCall(String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException { - return null; - } - - @Override - public PreparedStatement prepareStatement(String sql, int autoGeneratedKeys) throws SQLException { - return null; - } - - @Override - public PreparedStatement prepareStatement(String sql, int[] columnIndexes) throws SQLException { - return null; - } - - @Override - public PreparedStatement prepareStatement(String sql, String[] columnNames) throws SQLException { - return null; - } - - @Override - public Clob createClob() throws SQLException { - return null; - } - - @Override - public Blob createBlob() throws SQLException { - return null; - } - - @Override - public NClob createNClob() throws SQLException { - return null; - } - - @Override - public SQLXML createSQLXML() throws SQLException { - return null; - } - - @Override - public boolean isValid(int timeout) throws SQLException { - return false; - } - - @Override - public void setClientInfo(String name, String value) throws SQLClientInfoException { - - } - - @Override - public void setClientInfo(Properties properties) throws SQLClientInfoException { - - } - - @Override - public String getClientInfo(String name) throws SQLException { - return null; - } - - @Override - public Properties getClientInfo() throws SQLException { - return null; - } - - @Override - public Array createArrayOf(String typeName, Object[] elements) throws SQLException { - return null; - } - - @Override - public Struct createStruct(String typeName, Object[] attributes) throws SQLException { - return null; - } - - @Override - public void setSchema(String schema) throws SQLException { - - } - - @Override - public String getSchema() throws SQLException { - return null; - } - - @Override - public void abort(Executor executor) throws SQLException { - - } - - @Override - public void setNetworkTimeout(Executor executor, int milliseconds) throws SQLException { - - } - - @Override - public int getNetworkTimeout() throws SQLException { - return 0; - } - - @Override - public <T> T unwrap(Class<T> iface) throws SQLException { - return null; - } - - @Override - public boolean isWrapperFor(Class<?> iface) throws SQLException { - return false; - } -} + @Test + public void getPropertyKey() { + HiveInterpreter hi = new HiveInterpreter(new Properties()); + hi.open(); + String testCommand = "(default)\nshow tables"; + assertEquals("get key of default", "default", hi.getPropertyKey(testCommand)); + testCommand = "(default) show tables"; + assertEquals("get key of default", "default", hi.getPropertyKey(testCommand)); + hi.close(); + } +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java ---------------------------------------------------------------------- diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java index ef1f115..27f4b65 100644 --- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java +++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java @@ -193,6 +193,7 @@ public class RemoteInterpreter extends Interpreter { @Override public InterpreterResult interpret(String st, InterpreterContext context) { + logger.debug("st: {}", st); FormType form = getFormType(); RemoteInterpreterProcess interpreterProcess = getInterpreterProcess(); Client client = null; http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java ---------------------------------------------------------------------- diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java index 725ee63..46a0708 100644 --- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java +++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java @@ -203,6 +203,7 @@ public class RemoteInterpreterServer @Override public RemoteInterpreterResult interpret(String className, String st, RemoteInterpreterContext interpreterContext) throws TException { + logger.debug("st: {}", st); Interpreter intp = getInterpreter(className); InterpreterContext context = convert(interpreterContext); http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java ---------------------------------------------------------------------- diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java index 348c12e..ec47efd 100644 --- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java +++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java @@ -103,7 +103,7 @@ public class Paragraph extends Job implements Serializable, Cloneable { int scriptHeadIndex = 0; for (int i = 0; i < text.length(); i++) { char ch = text.charAt(i); - if (ch == ' ' || ch == '\n') { + if (ch == ' ' || ch == '\n' || ch == '(') { scriptHeadIndex = i; break; } @@ -132,10 +132,10 @@ public class Paragraph extends Job implements Serializable, Cloneable { if (magic == null) { return text; } - if (magic.length() + 2 >= text.length()) { + if (magic.length() + 1 >= text.length()) { return ""; } - return text.substring(magic.length() + 2); + return text.substring(magic.length() + 1).trim(); } public NoteInterpreterLoader getNoteReplLoader() { http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/79059395/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java ---------------------------------------------------------------------- diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java new file mode 100644 index 0000000..87805ce --- /dev/null +++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zeppelin.notebook; + +import org.junit.Test; + +import static org.junit.Assert.assertEquals; + +public class ParagraphTest { + @Test + public void scriptBodyWithReplName() { + String text = "%spark(1234567"; + assertEquals("(1234567", Paragraph.getScriptBody(text)); + + text = "%table 1234567"; + assertEquals("1234567", Paragraph.getScriptBody(text)); + } + @Test + public void scriptBodyWithoutReplName() { + String text = "12345678"; + assertEquals(text, Paragraph.getScriptBody(text)); + } +}
![https://cloud.githubusercontent.com/assets/3612566/11305933/835601e8-8ff2-11e5-9632-6fbbd97010c5.png"](https://cloud.githubusercontent.com/assets/3612566/11305933/835601e8-8ff2-11e5-9632-6fbbd97010c5.png"){kind=link}
![https://cloud.githubusercontent.com/assets/3612566/11306006/dd0882c4-8ff2-11e5-9094-e92dc762a7a6.png"](https://cloud.githubusercontent.com/assets/3612566/11306006/dd0882c4-8ff2-11e5-9094-e92dc762a7a6.png"){kind=link}