Okay, it is working with varchar columns only. Is there any way to workaround this?
On Fri, Jan 27, 2017 at 12:22 PM, ayan guha <guha.a...@gmail.com> wrote: > hi > > I thought so too, so I created a table with INT and Varchar columns > > desc agtest1 > > Name Null Type > ---- ---- ------------- > PID NUMBER(38) > DES VARCHAR2(100) > > url="jdbc:oracle:thin:@mpimpclu1-scan:1521/DEVAIM" > table = "agtest1" > user = "bal" > password= "bal" > driver="oracle.jdbc.OracleDriver" > df = sqlContext.read.jdbc(url=url,table=table,properties={"user" > :user,"password":password,"driver":driver}) > > > Still the issue persists. > > On Fri, Jan 27, 2017 at 11:19 AM, Takeshi Yamamuro <linguin....@gmail.com> > wrote: > >> Hi, >> >> I think you got this error because you used `NUMERIC` types in your >> schema (https://github.com/apache/spark/blob/master/sql/core/src/ >> main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala#L32). So, IIUC >> avoiding the type is a workaround. >> >> // maropu >> >> >> On Fri, Jan 27, 2017 at 8:18 AM, ayan guha <guha.a...@gmail.com> wrote: >> >>> Hi >>> >>> I am facing exact issue with Oracle/Exadataas mentioned here >>> <http://stackoverflow.com/questions/41873449/sparksql-key-not-found-scale>. >>> Any idea? I could not figure out so sending to this grou hoping someone >>> have see it (and solved it) >>> >>> Spark Version: 1.6 >>> pyspark command: >>> >>> pyspark --driver-class-path /opt/oracle/bigdatasql/bdcell- >>> 12.1/jlib-bds/kvclient.jar:/opt/oracle/bigdatasql/bdcell-12. >>> 1/jlib-bds/ojdbc7.jar:/opt/oracle/bigdatasql/bdcell-12.1/jli >>> b-bds/ojdbc7-orig.jar:/opt/oracle/bigdatasql/bdcell-12.1/jli >>> b-bds/oracle-hadoop-sql.jar:/opt/oracle/bigdatasql/bdcell-12 >>> .1/jlib-bds/ora-hadoop-common.jar:/opt/oracle/bigdatasql/ >>> bdcell-12.1/jlib-bds/ora-hadoop-common-orig.jar:/opt/oracle/ >>> bigdatasql/bdcell-12.1/jlib-bds/orahivedp.jar:/opt/oracle/ >>> bigdatasql/bdcell-12.1/jlib-bds/orahivedp-orig.jar:/opt/ >>> oracle/bigdatasql/bdcell-12.1/jlib-bds/orai18n.jar:/opt/ >>> oracle/bigdatasql/bdcell-12.1/jlib-bds/orai18n-orig.jar:/ >>> opt/oracle/bigdatasql/bdcell-12.1/jlib-bds/oraloader.jar:/ >>> opt/oracle/bigdatasql/bdcell-12.1/jlib-bds/oraloader-orig.jar --conf >>> spark.jars=/opt/oracle/bigdatasql/bdcell-12.1/jlib-bds/oracl >>> e-hadoop-sql.jar,/opt/oracle/bigdatasql/bdcell-12.1/jlib- >>> bds/ora-hadoop-common.jar,/opt/oracle/bigdatasql/bdcell- >>> 12.1/jlib-bds/orahivedp.jar,/opt/oracle/bigdatasql/bdcell- >>> 12.1/jlib-bds/oraloader.jar,/opt/oracle/bigdatasql/bdcell- >>> 12.1/jlib-bds/ojdbc7.jar,/opt/oracle/bigdatasql/bdcell-12.1/ >>> jlib-bds/orai18n.jar/opt/oracle/bigdatasql/bdcell-12.1/ >>> jlib-bds/kvclient.jar,/opt/oracle/bigdatasql/bdcell-12.1/ >>> jlib-bds/ojdbc7.jar,/opt/oracle/bigdatasql/bdcell-12.1/ >>> jlib-bds/ojdbc7-orig.jar,/opt/oracle/bigdatasql/bdcell-12.1/ >>> jlib-bds/oracle-hadoop-sql.jar,/opt/oracle/bigdatasql/ >>> bdcell-12.1/jlib-bds/ora-hadoop-common.jar,/opt/oracle/ >>> bigdatasql/bdcell-12.1/jlib-bds/ora-hadoop-common-orig. >>> jar,/opt/oracle/bigdatasql/bdcell-12.1/jlib-bds/ >>> orahivedp.jar,/opt/oracle/bigdatasql/bdcell-12.1/jlib- >>> bds/orahivedp-orig.jar,/opt/oracle/bigdatasql/bdcell-12.1/ >>> jlib-bds/orai18n.jar,/opt/oracle/bigdatasql/bdcell-12.1/ >>> jlib-bds/orai18n-orig.jar,/opt/oracle/bigdatasql/bdcell- >>> 12.1/jlib-bds/oraloader.jar,/opt/oracle/bigdatasql/bdcell- >>> 12.1/jlib-bds/oraloader-orig.jar >>> >>> >>> Here is my code: >>> >>> url="jdbc:oracle:thin:@mpimpclu1-scan:1521/DEVAIM" >>> table = "HIST_FORECAST_NEXT_BILL_DGTL" >>> user = "bal" >>> password= "bal" >>> driver="oracle.jdbc.OracleDriver" >>> df = sqlContext.read.jdbc(url=url,table=table,properties={"user": >>> user,"password":password,"driver":driver}) >>> >>> >>> Error: >>> Traceback (most recent call last): >>> File "<stdin>", line 1, in <module> >>> File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p2001.2081/lib/s >>> park/python/pyspark/sql/readwriter.py", line 289, in jdbc >>> return self._df(self._jreader.jdbc(url, table, jprop)) >>> File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p2001.2081/lib/s >>> park/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in >>> __call__ >>> File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p2001.2081/lib/s >>> park/python/pyspark/sql/utils.py", line 45, in deco >>> return f(*a, **kw) >>> File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p2001.2081/lib/s >>> park/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in >>> get_return_value >>> py4j.protocol.Py4JJavaError: An error occurred while calling o40.jdbc. >>> : java.util.NoSuchElementException: key not found: scale >>> at scala.collection.MapLike$class.default(MapLike.scala:228) >>> at scala.collection.AbstractMap.default(Map.scala:58) >>> at scala.collection.MapLike$class.apply(MapLike.scala:141) >>> at scala.collection.AbstractMap.apply(Map.scala:58) >>> at org.apache.spark.sql.types.Metadata.get(Metadata.scala:108) >>> at org.apache.spark.sql.types.Metadata.getLong(Metadata.scala:5 >>> 1) >>> at org.apache.spark.sql.jdbc.OracleDialect$.getCatalystType(Ora >>> cleDialect.scala:33) >>> at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.res >>> olveTable(JDBCRDD.scala:140) >>> at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation >>> .<init>(JDBCRelation.scala:91) >>> at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.sc >>> ala:222) >>> at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.sc >>> ala:146) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>> ssorImpl.java:62) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) >>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.jav >>> a:381) >>> at py4j.Gateway.invoke(Gateway.java:259) >>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.j >>> ava:133) >>> at py4j.commands.CallCommand.execute(CallCommand.java:79) >>> at py4j.GatewayConnection.run(GatewayConnection.java:209) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> >>> -- >>> Best Regards, >>> Ayan Guha >>> >> >> >> >> -- >> --- >> Takeshi Yamamuro >> > > > > -- > Best Regards, > Ayan Guha > -- Best Regards, Ayan Guha