Re: Bit(N) on create Table with MSSQLServer

[Mich Talebzadeh]

Hang on

 Are you talking about target database in MSSQL created and dropped?

Is your Spark JDBC credential in MSSQL a DBO or sa?



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 4 May 2016 at 20:54, Andrés Ivaldi  wrote:

> Ok, so I did that, create a database and then insert data, but spark drop
> database and try to create it again, I'm using
> Dataframe.write(SaveMode.Overwrite), documentation said :
> "when performing a Overwrite, the data will be deleted before writing out
> the new data."
>
> why is dropping the table?
>
>
> On Wed, May 4, 2016 at 6:44 AM, Andrés Ivaldi  wrote:
>
>> Yes, I can do that, it's what we are doing now, but I think the best
>> approach would be delegate the create table action to spark.
>>
>> On Tue, May 3, 2016 at 8:17 PM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Can you create the MSSQL (target) table first with the correct column
>>> setting and insert data from Spark to it with JDBC as opposed to JDBC
>>> creating target table itself?
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 3 May 2016 at 22:19, Andrés Ivaldi  wrote:
>>>
 Ok, Spark MSSQL dataType mapping is not right for me, ie. string is
 Text instead of varchar(MAX) , so how can I override default SQL Mapping?

 regards.

 On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> Well if MSSQL cannot create that column then it is more like
> compatibility between Spark and RDBMS.
>
> What value that column has in MSSQL. Can you create table the table in
> MSSQL database or map it in Spark to a valid column before opening JDBC
> connection?
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 29 April 2016 at 16:16, Andrés Ivaldi  wrote:
>
>> Hello, Spark is executing a create table sentence (using JDBC) to
>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>> types, This create table cannot be executed on MSSQLServer.
>>
>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>> question is, this is a problem of spark or JDBC driver who is not mapping
>> right?
>>
>> Anyway it´s possible to override that mapping in Spark?
>>
>> Regards
>>
>> --
>> Ing. Ivaldi Andres
>>
>
>


 --
 Ing. Ivaldi Andres

>>>
>>>
>>
>>
>> --
>> Ing. Ivaldi Andres
>>
>
>
>
> --
> Ing. Ivaldi Andres
>

Re: Bit(N) on create Table with MSSQLServer

[Andrés Ivaldi]

Ok, so I did that, create a database and then insert data, but spark drop
database and try to create it again, I'm using
Dataframe.write(SaveMode.Overwrite), documentation said :
"when performing a Overwrite, the data will be deleted before writing out
the new data."

why is dropping the table?


On Wed, May 4, 2016 at 6:44 AM, Andrés Ivaldi  wrote:

> Yes, I can do that, it's what we are doing now, but I think the best
> approach would be delegate the create table action to spark.
>
> On Tue, May 3, 2016 at 8:17 PM, Mich Talebzadeh  > wrote:
>
>> Can you create the MSSQL (target) table first with the correct column
>> setting and insert data from Spark to it with JDBC as opposed to JDBC
>> creating target table itself?
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 3 May 2016 at 22:19, Andrés Ivaldi  wrote:
>>
>>> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
>>> instead of varchar(MAX) , so how can I override default SQL Mapping?
>>>
>>> regards.
>>>
>>> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 Well if MSSQL cannot create that column then it is more like
 compatibility between Spark and RDBMS.

 What value that column has in MSSQL. Can you create table the table in
 MSSQL database or map it in Spark to a valid column before opening JDBC
 connection?

 HTH

 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



 On 29 April 2016 at 16:16, Andrés Ivaldi  wrote:

> Hello, Spark is executing a create table sentence (using JDBC) to
> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
> types, This create table cannot be executed on MSSQLServer.
>
> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
> question is, this is a problem of spark or JDBC driver who is not mapping
> right?
>
> Anyway it´s possible to override that mapping in Spark?
>
> Regards
>
> --
> Ing. Ivaldi Andres
>


>>>
>>>
>>> --
>>> Ing. Ivaldi Andres
>>>
>>
>>
>
>
> --
> Ing. Ivaldi Andres
>



-- 
Ing. Ivaldi Andres

Re: Bit(N) on create Table with MSSQLServer

[Andrés Ivaldi]

Yes, I can do that, it's what we are doing now, but I think the best
approach would be delegate the create table action to spark.

On Tue, May 3, 2016 at 8:17 PM, Mich Talebzadeh 
wrote:

> Can you create the MSSQL (target) table first with the correct column
> setting and insert data from Spark to it with JDBC as opposed to JDBC
> creating target table itself?
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 3 May 2016 at 22:19, Andrés Ivaldi  wrote:
>
>> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
>> instead of varchar(MAX) , so how can I override default SQL Mapping?
>>
>> regards.
>>
>> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Well if MSSQL cannot create that column then it is more like
>>> compatibility between Spark and RDBMS.
>>>
>>> What value that column has in MSSQL. Can you create table the table in
>>> MSSQL database or map it in Spark to a valid column before opening JDBC
>>> connection?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 29 April 2016 at 16:16, Andrés Ivaldi  wrote:
>>>
 Hello, Spark is executing a create table sentence (using JDBC) to
 MSSQLServer with a mapping column type like ColName Bit(1) for boolean
 types, This create table cannot be executed on MSSQLServer.

 In class JdbcDialect the mapping for Boolean type is Bit(1), so the
 question is, this is a problem of spark or JDBC driver who is not mapping
 right?

 Anyway it´s possible to override that mapping in Spark?

 Regards

 --
 Ing. Ivaldi Andres

>>>
>>>
>>
>>
>> --
>> Ing. Ivaldi Andres
>>
>
>


-- 
Ing. Ivaldi Andres

Re: Bit(N) on create Table with MSSQLServer

[Mich Talebzadeh]

Can you create the MSSQL (target) table first with the correct column
setting and insert data from Spark to it with JDBC as opposed to JDBC
creating target table itself?



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 3 May 2016 at 22:19, Andrés Ivaldi  wrote:

> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
> instead of varchar(MAX) , so how can I override default SQL Mapping?
>
> regards.
>
> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh  > wrote:
>
>> Well if MSSQL cannot create that column then it is more like
>> compatibility between Spark and RDBMS.
>>
>> What value that column has in MSSQL. Can you create table the table in
>> MSSQL database or map it in Spark to a valid column before opening JDBC
>> connection?
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 29 April 2016 at 16:16, Andrés Ivaldi  wrote:
>>
>>> Hello, Spark is executing a create table sentence (using JDBC) to
>>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>>> types, This create table cannot be executed on MSSQLServer.
>>>
>>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>>> question is, this is a problem of spark or JDBC driver who is not mapping
>>> right?
>>>
>>> Anyway it´s possible to override that mapping in Spark?
>>>
>>> Regards
>>>
>>> --
>>> Ing. Ivaldi Andres
>>>
>>
>>
>
>
> --
> Ing. Ivaldi Andres
>

Re: Bit(N) on create Table with MSSQLServer

[Mich Talebzadeh]

Well if MSSQL cannot create that column then it is more like compatibility
between Spark and RDBMS.

What value that column has in MSSQL. Can you create table the table in
MSSQL database or map it in Spark to a valid column before opening JDBC
connection?

HTH

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 29 April 2016 at 16:16, Andrés Ivaldi  wrote:

> Hello, Spark is executing a create table sentence (using JDBC) to
> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
> types, This create table cannot be executed on MSSQLServer.
>
> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
> question is, this is a problem of spark or JDBC driver who is not mapping
> right?
>
> Anyway it´s possible to override that mapping in Spark?
>
> Regards
>
> --
> Ing. Ivaldi Andres
>

Bit(N) on create Table with MSSQLServer

[Andrés Ivaldi]

Hello, Spark is executing a create table sentence (using JDBC) to
MSSQLServer with a mapping column type like ColName Bit(1) for boolean
types, This create table cannot be executed on MSSQLServer.

In class JdbcDialect the mapping for Boolean type is Bit(1), so the
question is, this is a problem of spark or JDBC driver who is not mapping
right?

Anyway it´s possible to override that mapping in Spark?

Regards

-- 
Ing. Ivaldi Andres