Well, actually the logic is to look for whether you have specified the host in the extra params, and if you haven’t — set host to ‘NoHostProvided’ (the default)
Great that you are willing to take a stab at that! -Robin > On 24 Mar 2017, at 11:16, Jorge Alpedrinha Ramos <[email protected]> > wrote: > > Thank you Robin, > > Your code provides a simple solution for the problem, although I believe > that forcing host to be defined may be unnecessary because most use cases > won't require it. After looking at the host hack used with boto I really > think S3Hook should start using boto3, the fact that you can't pass a > region string such as *us-east-1* using host instead doesn't seem standard > and is not reusable for any other purpose (other than s3). > > I think https://issues.apache.org/jira/browse/AIRFLOW-115 is the key point. > I'll look into making a proposition on the matter. > > Thank you all for your help. > > > On Thu, Mar 23, 2017 at 5:16 PM Gael Magnan <[email protected]> wrote: > >> Hi, >> >> we have had the same problem and basically did the same as Robin, the only >> diff we have is we use the host from the connection instead of extra >> params. >> >> I can't agree more with the statement that we need to really rethink the >> S3Hook, and maybe make some basic S3/AWS operators. >> >> Gael >> >> Le jeu. 23 mars 2017 à 18:05, Robin Bartholdson < >> [email protected]> a écrit : >> >>> I submitted our patch in the mean time, do test it and see if it works >> for >>> you :-) >>> >>> https://github.com/apache/incubator-airflow/pull/2181 >>> https://issues.apache.org/jira/browse/AIRFLOW-1034 >>> >>> -Robin >>> >>> On 23 Mar 2017, at 17:43, Robin Bartholdson < >>> [email protected]> wrote: >>> >>> There is some discussion in this JIRA, >>> https://issues.apache.org/jira/browse/AIRFLOW-115, about moving to boto3 >>> and restructuring along the lines you discuss. >>> >>> I totally agree that it would indeed be very useful! >>> >>> -Robin >>> >>> >>> On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos < >>> [email protected]> wrote: >>> >>> Hi Sumit, >>> >>> I was actually referring to what Robin (thank you) pointed out which is >>> reason enough for having region in S3Hook. >>> >>> There are even other situations having the region would be beneficial, >>> that's why the AwsHook has that definition, although I would agree that >>> their goals are different they typically refer to the same credentials, >>> just different use cases. >>> >>> This makes me feel that it becomes redundant to have both an S3 >> connection >>> and an AWS connection with the same credentials just be able to access >>> region. Wouldn't it be beneficial to implement S3Hook (and S3 connection) >>> as a special case of AwsHook (and Aws connection) and be able to use both >>> interchangeably ? >>> >>> I'm also aware that one uses boto and the other uses boto3, and it seems >>> like the project is naturally slowly migrating towards the later. Maybe >>> there is even some discussion regarding this that I'm not aware. >>> >>> If you think this is a desirable move, I may start to design this move, >>> making sure nothing gets broken in the process. >>> >>> WDYT? >>> >>> On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson < >>> [email protected]> wrote: >>> >>> I _think_ that’s a different issue (do correct me if I’m wrong) >>> >>> For a number of regions (see the list on >>> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html), >>> you have to specify an extra host-parameter in the boto-S3Connection as >>> they only support signature V4 and the host-param is required to have >> this >>> working ( >>> >>> >> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError >>> ) >>> >>> -Robin >>> >>> On 23 Mar 2017, at 15:10, Sumit Maheshwari <[email protected]> >> wrote: >>> >>> Are we talking about this >>> https://github.com/apache/incubator-airflow/pull/2176 ? >>> >>> >>> >>> >>> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson < >>> [email protected]> wrote: >>> >>> Hi, >>> >>> We actually have a patch for this that allows you to specify the host >>> parameter in the S3 config in the connections. >>> >>> Been thinking about submitting this for a while but have not had the >> time, >>> maybe now’s the time to do it :-) >>> >>> If you want to open a JIRA I can submit the patch we are using in >>> production! >>> >>> -Robin >>> >>> >>> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos < >>> [email protected]> wrote: >>> >>> Hi, >>> >>> I've been using S3Hook for a while, but I've run into some issues for not >>> having a defined region. I can't really say why I can't access this >> bucket >>> without defining the region, but I know that there is no way to define >>> region in S3Hook via the connection extra parameter unless a config_file >> is >>> used, which is not always an available method. >>> >>> I haven't found any ticket regarding this, is there any alternative ? >>> >>> Regards, >>> Jorge >>> >>> >>> >>> >>> >>> >>> >>> >>
smime.p7s
Description: S/MIME cryptographic signature
