Re: Re: Re: Re: Re: Integrate PyHive and Alembic

2020-03-02 Thread Ke Zhu - k...@us.ibm.com
Mike, Thanks for merging my change on Alembic! On Thu, 2020-02-27 at 15:38 -0500, Mike Bayer wrote: > > > On Thu, Feb 27, 2020, at 2:49 PM, Ke Zhu - k...@us.ibm.com wrote: > > On Wed, 2020-02-26 at 11:07 -0500, Mike Bayer wrote: > > > oh, that issue is talking about rowcount. Alembic does not

Re: Re: Re: Re: Integrate PyHive and Alembic

2020-02-27 Thread Mike Bayer
On Thu, Feb 27, 2020, at 2:49 PM, Ke Zhu - k...@us.ibm.com wrote: > On Wed, 2020-02-26 at 11:07 -0500, Mike Bayer wrote: >> oh, that issue is talking about rowcount. Alembic does not need rowcount to >> function correctly. I see that Alembic is doing this now, however there is a >> dialect-leve

Re: Re: Re: Re: Integrate PyHive and Alembic

2020-02-27 Thread Ke Zhu - k...@us.ibm.com
On Wed, 2020-02-26 at 11:07 -0500, Mike Bayer wrote: oh, that issue is talking about rowcount. Alembic does not need rowcount to function correctly.I see that Alembic is doing this now, however there is a dialect-level flag called "supports_sane_rowcount", if this were False, Alembic should

Re: Re: Re: Integrate PyHive and Alembic

2020-02-26 Thread Mike Bayer
On Tue, Feb 25, 2020, at 11:06 AM, Ke Zhu - k...@us.ibm.com wrote: > According to Apache Hive, Update/Delete can only be performed on tables that > supports ACID. see > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Update > > > this is a database that c

Re: Re: Re: Integrate PyHive and Alembic

2020-02-25 Thread Ke Zhu - k...@us.ibm.com
According to Apache Hive, Update/Delete can only be performed on tables that supports ACID. see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Update > this is a database that can change schema structures but not > insert/update/delete rows? is there some

Re: Re: Integrate PyHive and Alembic

2020-02-24 Thread Mike Bayer
On Mon, Feb 24, 2020, at 3:44 PM, Ke Zhu - k...@us.ibm.com wrote: > Mike, > > Thanks for the pointers. I've figured out the programming part and discovers > more things when integrating PyHive: > > 1. It requires the table alembic_version to be transactional so that it can > do Update/Delete

Re: Re: Integrate PyHive and Alembic

2020-02-24 Thread Ke Zhu - k...@us.ibm.com
Mike, Thanks for the pointers. I've figured out the programming part and discovers more things when integrating PyHive: 1. It requires the table alembic_version to be transactional so that it can do Update/Delete when upgrade/downgrade. which is challege for Hive3 which has limited ACID support. Si

Re: Integrate PyHive and Alembic

2020-01-25 Thread Mike Bayer
On Fri, Jan 24, 2020, at 1:56 PM, Ke Zhu wrote: > Just discovered this post when trying to do exact same thing (besides > planning to support one more dialect). > > > Anywhere in your hive dialect, simply put the above code that you have > > (using the correct imports of course). > > Does it

Re: Integrate PyHive and Alembic

2020-01-24 Thread Ke Zhu
Just discovered this post when trying to do exact same thing (besides planning to support one more dialect). > Anywhere in your hive dialect, simply put the above code that you have (using the correct imports of course). Does it mean it must introduce dependency to alembic (since it uses ale

Re: Integrate PyHive and Alembic

2017-03-01 Thread Alexander Peletz
The dialect subclassed the SQLCompiler but not the DDLCompiler. Putting the visit_* statements in a DDLCompiler subclass worked! Thanks! Now to override the UPDATE and DELETE statements since Hive doesn't support those... Will give it a whirl before asking more questions. Thanks, Alexander On

Re: Integrate PyHive and Alembic

2017-03-01 Thread mike bayer
does your dialect override DDLCompiler.get_column_specification()? That's the best way to get in extra things like NOT NULL, etc. Example: https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/dialects/mysql/base.py#L982 On 03/01/2017 02:13 PM, Alexander Peletz wrote: I am using t

Re: Integrate PyHive and Alembic

2017-03-01 Thread Alexander Peletz
I am using the pyhive library which includes a partial implementation of the 'hive' dialect. I have edited various aspects of this dialect but I am still unable to affect the initial create table statement that Alembic is issuing to build the alembic_version table. Here is what Alembic is issuin

Re: Integrate PyHive and Alembic

2017-02-10 Thread mike bayer
On 02/10/2017 12:41 PM, Alexander Peletz wrote: Great thank you for the quick response! I actually found this morning that I can put the class definition in alembic/ddl/impl.py and it works (seemed like a logical module for it?). Next issue is the CREATE TABLE statement used to create the alem

Re: Integrate PyHive and Alembic

2017-02-10 Thread Alexander Peletz
Great thank you for the quick response! I actually found this morning that I can put the class definition in alembic/ddl/impl.py and it works (seemed like a logical module for it?). Next issue is the CREATE TABLE statement used to create the alembic_version table uses keywords that are not vali

Re: Integrate PyHive and Alembic

2017-02-10 Thread mike bayer
On 02/10/2017 07:41 AM, Alexander Peletz wrote: Hello, I would like to use Alembic to manage my Hive Metastore. I have installed, PyHive, SqlAlchemy, and Alembic. I am able to create a functional engine object using the 'hive' dialect in sqlalchemy, however I cannot get Alembic to recognize th

Integrate PyHive and Alembic

2017-02-10 Thread Alexander Peletz
Hello, I would like to use Alembic to manage my Hive Metastore. I have installed, PyHive, SqlAlchemy, and Alembic. I am able to create a functional engine object using the 'hive' dialect in sqlalchemy, however I cannot get Alembic to recognize this dialect. The problem appears to be a lack of a