Hi, we have had the same problem and basically did the same as Robin, the only diff we have is we use the host from the connection instead of extra params.
I can't agree more with the statement that we need to really rethink the S3Hook, and maybe make some basic S3/AWS operators. Gael Le jeu. 23 mars 2017 à 18:05, Robin Bartholdson < [email protected]> a écrit : > I submitted our patch in the mean time, do test it and see if it works for > you :-) > > https://github.com/apache/incubator-airflow/pull/2181 > https://issues.apache.org/jira/browse/AIRFLOW-1034 > > -Robin > > On 23 Mar 2017, at 17:43, Robin Bartholdson < > [email protected]> wrote: > > There is some discussion in this JIRA, > https://issues.apache.org/jira/browse/AIRFLOW-115, about moving to boto3 > and restructuring along the lines you discuss. > > I totally agree that it would indeed be very useful! > > -Robin > > > On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos < > [email protected]> wrote: > > Hi Sumit, > > I was actually referring to what Robin (thank you) pointed out which is > reason enough for having region in S3Hook. > > There are even other situations having the region would be beneficial, > that's why the AwsHook has that definition, although I would agree that > their goals are different they typically refer to the same credentials, > just different use cases. > > This makes me feel that it becomes redundant to have both an S3 connection > and an AWS connection with the same credentials just be able to access > region. Wouldn't it be beneficial to implement S3Hook (and S3 connection) > as a special case of AwsHook (and Aws connection) and be able to use both > interchangeably ? > > I'm also aware that one uses boto and the other uses boto3, and it seems > like the project is naturally slowly migrating towards the later. Maybe > there is even some discussion regarding this that I'm not aware. > > If you think this is a desirable move, I may start to design this move, > making sure nothing gets broken in the process. > > WDYT? > > On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson < > [email protected]> wrote: > > I _think_ that’s a different issue (do correct me if I’m wrong) > > For a number of regions (see the list on > http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html), > you have to specify an extra host-parameter in the boto-S3Connection as > they only support signature V4 and the host-param is required to have this > working ( > > http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError > ) > > -Robin > > On 23 Mar 2017, at 15:10, Sumit Maheshwari <[email protected]> wrote: > > Are we talking about this > https://github.com/apache/incubator-airflow/pull/2176 ? > > > > > On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson < > [email protected]> wrote: > > Hi, > > We actually have a patch for this that allows you to specify the host > parameter in the S3 config in the connections. > > Been thinking about submitting this for a while but have not had the time, > maybe now’s the time to do it :-) > > If you want to open a JIRA I can submit the patch we are using in > production! > > -Robin > > > On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos < > [email protected]> wrote: > > Hi, > > I've been using S3Hook for a while, but I've run into some issues for not > having a defined region. I can't really say why I can't access this bucket > without defining the region, but I know that there is no way to define > region in S3Hook via the connection extra parameter unless a config_file is > used, which is not always an available method. > > I haven't found any ticket regarding this, is there any alternative ? > > Regards, > Jorge > > > > > > > >
