I submitted our patch in the mean time, do test it and see if it works for you :-)
https://github.com/apache/incubator-airflow/pull/2181 https://issues.apache.org/jira/browse/AIRFLOW-1034 -Robin > On 23 Mar 2017, at 17:43, Robin Bartholdson <[email protected]> > wrote: > > There is some discussion in this JIRA, > https://issues.apache.org/jira/browse/AIRFLOW-115 > <https://issues.apache.org/jira/browse/AIRFLOW-115>, about moving to boto3 > and restructuring along the lines you discuss. > > I totally agree that it would indeed be very useful! > > -Robin > > >> On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Sumit, >> >> I was actually referring to what Robin (thank you) pointed out which is >> reason enough for having region in S3Hook. >> >> There are even other situations having the region would be beneficial, >> that's why the AwsHook has that definition, although I would agree that >> their goals are different they typically refer to the same credentials, >> just different use cases. >> >> This makes me feel that it becomes redundant to have both an S3 connection >> and an AWS connection with the same credentials just be able to access >> region. Wouldn't it be beneficial to implement S3Hook (and S3 connection) >> as a special case of AwsHook (and Aws connection) and be able to use both >> interchangeably ? >> >> I'm also aware that one uses boto and the other uses boto3, and it seems >> like the project is naturally slowly migrating towards the later. Maybe >> there is even some discussion regarding this that I'm not aware. >> >> If you think this is a desirable move, I may start to design this move, >> making sure nothing gets broken in the process. >> >> WDYT? >> >> On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson < >> [email protected] <mailto:[email protected]>> >> wrote: >> >>> I _think_ that’s a different issue (do correct me if I’m wrong) >>> >>> For a number of regions (see the list on >>> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html >>> <http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html>), >>> you have to specify an extra host-parameter in the boto-S3Connection as >>> they only support signature V4 and the host-param is required to have this >>> working ( >>> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError >>> >>> <http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError> >>> ) >>> >>> -Robin >>> >>> On 23 Mar 2017, at 15:10, Sumit Maheshwari <[email protected]> wrote: >>> >>> Are we talking about this >>> https://github.com/apache/incubator-airflow/pull/2176 ? >>> >>> >>> >>> >>> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson < >>> [email protected]> wrote: >>> >>> Hi, >>> >>> We actually have a patch for this that allows you to specify the host >>> parameter in the S3 config in the connections. >>> >>> Been thinking about submitting this for a while but have not had the time, >>> maybe now’s the time to do it :-) >>> >>> If you want to open a JIRA I can submit the patch we are using in >>> production! >>> >>> -Robin >>> >>> >>> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos < >>> [email protected]> wrote: >>> >>> Hi, >>> >>> I've been using S3Hook for a while, but I've run into some issues for not >>> having a defined region. I can't really say why I can't access this bucket >>> without defining the region, but I know that there is no way to define >>> region in S3Hook via the connection extra parameter unless a config_file is >>> used, which is not always an available method. >>> >>> I haven't found any ticket regarding this, is there any alternative ? >>> >>> Regards, >>> Jorge >>> >>> >>> >>> >>> >
smime.p7s
Description: S/MIME cryptographic signature
